[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:17:15 dist-test-slave-h37r krb5kdc[4666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:17:15 dist-test-slave-h37r krb5kdc[4666](info): set up 2 sockets
Sep 28 02:17:15 dist-test-slave-h37r krb5kdc[4666](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 02:17:16.496260 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.008s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:17:16 dist-test-slave-h37r krb5kdc[4666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:17:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:17:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:17:16.672788 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:43351
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:33433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:43351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:17.001391 4679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:17.001751 4679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:17.001920 4679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:17.009204 4679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:17:17.009373 4679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:17.009486 4679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:17.009590 4679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:17:17.009682 4679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:17:17.019686 4679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:33433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:43351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:43351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:17.021106 4679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:17.022466 4679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:17.035231 4687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:17.035382 4684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:17.035360 4685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:17.035882 4679 server_base.cc:1034] running on GCE node
I20260928 02:17:17.036886 4679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:17.038287 4679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:17.039548 4679 hybrid_clock.cc:648] HybridClock initialized: now 1790561837039506 us; error 56 us; skew 500 ppm
I20260928 02:17:17.040014 4679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:17 dist-test-slave-h37r krb5kdc[4666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:17.043476 4679 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:17:17.045390 4679 webserver.cc:460] Webserver started at http://127.4.140.254:37387/ using document root <none> and password file <none>
I20260928 02:17:17.046208 4679 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:17.046370 4679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:17.046715 4679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:17.050024 4679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "a9f8a9552e1b45fa91b67f7d9aa73700"
format_stamp: "Formatted at 2026-09-28 02:17:17 on dist-test-slave-h37r"
server_key: "5ddd3e2947479ec8145ddb90884aef95"
server_key_iv: "f99aa89fd6573940265de1ea42640d61"
server_key_version: "encryptionkey@0"
I20260928 02:17:17.050895 4679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "a9f8a9552e1b45fa91b67f7d9aa73700"
format_stamp: "Formatted at 2026-09-28 02:17:17 on dist-test-slave-h37r"
server_key: "5ddd3e2947479ec8145ddb90884aef95"
server_key_iv: "f99aa89fd6573940265de1ea42640d61"
server_key_version: "encryptionkey@0"
I20260928 02:17:17.056653 4679 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:17:17.060427 4694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:17.061859 4679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:17:17.062057 4679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "a9f8a9552e1b45fa91b67f7d9aa73700"
format_stamp: "Formatted at 2026-09-28 02:17:17 on dist-test-slave-h37r"
server_key: "5ddd3e2947479ec8145ddb90884aef95"
server_key_iv: "f99aa89fd6573940265de1ea42640d61"
server_key_version: "encryptionkey@0"
I20260928 02:17:17.062306 4679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:17.087538 4679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:17.092779 4679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:17.093114 4679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:17.113431 4679 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:43351
I20260928 02:17:17.113448 4745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:43351 every 8 connection(s)
I20260928 02:17:17.114986 4679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:17:17.118934 4746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:17.120213 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 4679
I20260928 02:17:17.120558 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:17:17.122967 4659 external_mini_cluster.cc:1428] Setting key 77f714036d6db4e23e77f1baa260c5bf
I20260928 02:17:17.132390 4746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700: Bootstrap starting.
I20260928 02:17:17.135434 4746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:17.136545 4746 log.cc:824] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:17.140157 4746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700: No bootstrap required, opened a new log
I20260928 02:17:17.148770 4746 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f8a9552e1b45fa91b67f7d9aa73700" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43351 } }
I20260928 02:17:17.149297 4746 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:17.149405 4746 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f8a9552e1b45fa91b67f7d9aa73700, State: Initialized, Role: FOLLOWER
I20260928 02:17:17.150143 4746 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f8a9552e1b45fa91b67f7d9aa73700" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43351 } }
I20260928 02:17:17.150399 4746 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:17:17.150539 4746 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:17:17.150748 4746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:17.153035 4746 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f8a9552e1b45fa91b67f7d9aa73700" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43351 } }
Sep 28 02:17:17 dist-test-slave-h37r krb5kdc[4666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:17.153646 4746 leader_election.cc:302] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9f8a9552e1b45fa91b67f7d9aa73700; no voters:
I20260928 02:17:17.154277 4746 leader_election.cc:288] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:17:17.154433 4751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:17.155542 4751 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700 [term 1 LEADER]: Becoming Leader. State: Replica: a9f8a9552e1b45fa91b67f7d9aa73700, State: Running, Role: LEADER
I20260928 02:17:17.156185 4751 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f8a9552e1b45fa91b67f7d9aa73700" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43351 } }
I20260928 02:17:17.156646 4746 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:17:17.162060 4752 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9f8a9552e1b45fa91b67f7d9aa73700" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f8a9552e1b45fa91b67f7d9aa73700" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43351 } } }
I20260928 02:17:17.162461 4752 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700 [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:17.162943 4753 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9f8a9552e1b45fa91b67f7d9aa73700. Latest consensus state: current_term: 1 leader_uuid: "a9f8a9552e1b45fa91b67f7d9aa73700" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f8a9552e1b45fa91b67f7d9aa73700" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43351 } } }
I20260928 02:17:17.163257 4753 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700 [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:17.166538 4760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:17:17.168673 4748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:17.127848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36744 (local address 127.4.140.254:43351)
0928 02:17:17.128852 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:17.128869 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:17.130219 (+ 1350us) server_negotiation.cc:407] Connection header received
0928 02:17:17.140002 (+ 9783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:17.140027 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:17.140517 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:17.141013 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:17.143793 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:17.145984 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:17.147745 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:17.148806 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:17.148992 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:17.155594 (+ 6602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:17.155607 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:17.155631 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:17.155698 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:17.160758 (+ 5060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:17.162154 (+ 1396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:17.162167 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:17.162179 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:17.162482 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:17.163428 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:17.163434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:17.163438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:17.164358 (+ 920us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:17.164746 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:17.166977 (+ 2231us) server_negotiation.cc:299] Negotiation successful
0928 02:17:17.167429 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":429,"threads_started":1}
I20260928 02:17:17.174685 4760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:17:17.183882 4760 catalog_manager.cc:1347] Generated new cluster ID: dab68280cb3944f7abd112676e539da5
I20260928 02:17:17.184073 4760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:17:17.204596 4760 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:17:17.206621 4760 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:17:17.215494 4760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9f8a9552e1b45fa91b67f7d9aa73700: Generated new TSK 0
I20260928 02:17:17.216617 4760 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
I20260928 02:17:17.265077 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43351
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:33433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:17.593321 4773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:17.593667 4773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:17.593852 4773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:17.601255 4773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:17.601418 4773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:17.601600 4773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:17:17.611444 4773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:33433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43351
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:17.612792 4773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:17.614187 4773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:17.627446 4778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:17.627467 4779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:17.627575 4781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:17.628690 4773 server_base.cc:1034] running on GCE node
I20260928 02:17:17.629294 4773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:17.630326 4773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:17.631531 4773 hybrid_clock.cc:648] HybridClock initialized: now 1790561837631482 us; error 42 us; skew 500 ppm
I20260928 02:17:17.631873 4773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:17 dist-test-slave-h37r krb5kdc[4666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:17.635142 4773 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:17:17.636711 4773 webserver.cc:460] Webserver started at http://127.4.140.193:35605/ using document root <none> and password file <none>
I20260928 02:17:17.637372 4773 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:17.637472 4773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:17.637715 4773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:17.640687 4773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "3001298e2ba74179a2f2de599de816dd"
format_stamp: "Formatted at 2026-09-28 02:17:17 on dist-test-slave-h37r"
server_key: "0fd85b49032645c8dd3fd09487921a9c"
server_key_iv: "c554617d5391081d78a0be4c5e318eff"
server_key_version: "encryptionkey@0"
I20260928 02:17:17.641458 4773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "3001298e2ba74179a2f2de599de816dd"
format_stamp: "Formatted at 2026-09-28 02:17:17 on dist-test-slave-h37r"
server_key: "0fd85b49032645c8dd3fd09487921a9c"
server_key_iv: "c554617d5391081d78a0be4c5e318eff"
server_key_version: "encryptionkey@0"
I20260928 02:17:17.646705 4773 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 02:17:17.650184 4788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:17.651724 4773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:17:17.651974 4773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "3001298e2ba74179a2f2de599de816dd"
format_stamp: "Formatted at 2026-09-28 02:17:17 on dist-test-slave-h37r"
server_key: "0fd85b49032645c8dd3fd09487921a9c"
server_key_iv: "c554617d5391081d78a0be4c5e318eff"
server_key_version: "encryptionkey@0"
I20260928 02:17:17.652227 4773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:17.677609 4773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:17.682173 4773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:17.682415 4773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:17.683364 4773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:17.684751 4773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:17.684825 4773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:17.684906 4773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:17.684948 4773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:17.718981 4773 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:36185
I20260928 02:17:17.719091 4900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:36185 every 8 connection(s)
I20260928 02:17:17.720448 4773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:17:17.722679 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 4773
I20260928 02:17:17.722906 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:17:17.723712 4659 external_mini_cluster.cc:1428] Setting key 25f27163290c6fe2f715fabeadb830b6
Sep 28 02:17:17 dist-test-slave-h37r krb5kdc[4666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.140.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.194@KRBTEST.COM" created.
I20260928 02:17:17.744766 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:17.723196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:44579 (local address 127.4.140.254:43351)
0928 02:17:17.723688 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:17.723694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:17.725117 (+ 1423us) server_negotiation.cc:407] Connection header received
0928 02:17:17.727666 (+ 2549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:17.727670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:17.727737 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:17.727831 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:17.730149 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:17.730909 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:17.733609 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:17.734083 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:17.734278 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:17.738659 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:17.738663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:17.738667 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:17.738700 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:17.741468 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:17.742298 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:17.742308 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:17.742310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:17.742423 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:17.742908 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:17.742912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:17.742915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:17.743176 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:17.743289 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:17.744087 (+ 798us) server_negotiation.cc:299] Negotiation successful
0928 02:17:17.744312 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":243,"threads_started":1}
I20260928 02:17:17.745795 4904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:17.724074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43351 (local address 127.4.140.193:44579)
0928 02:17:17.724943 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:17.724997 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:17.727339 (+ 2342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:17.728163 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:17.728179 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:17.729074 (+ 895us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:17.729928 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:17.729957 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:17.731073 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:17.731082 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:17.733425 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:17.733443 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:17.734407 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:17.734417 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:17.734768 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:17.735604 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:17.735638 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:17.738396 (+ 2758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:17.741628 (+ 3232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:17.741637 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:17.741656 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:17.742161 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:17.742535 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:17.742540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:17.742544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:17.742746 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:17.743295 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:17.743301 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:17.743617 (+ 316us) client_negotiation.cc:769] Sending connection context
0928 02:17:17.743934 (+ 317us) client_negotiation.cc:240] Negotiation successful
0928 02:17:17.744388 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":324,"threads_started":1}
I20260928 02:17:17.747437 4901 heartbeater.cc:347] Connected to a master server at 127.4.140.254:43351
I20260928 02:17:17.747754 4901 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:17.748534 4901 heartbeater.cc:511] Master 127.4.140.254:43351 requested a full tablet report, sending...
I20260928 02:17:17.750749 4711 ts_manager.cc:194] Registered new tserver with Master: 3001298e2ba74179a2f2de599de816dd (127.4.140.193:36185)
I20260928 02:17:17.752324 4711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:44579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
I20260928 02:17:17.767804 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43351
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:33433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:18.092310 4907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:18.092609 4907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:18.092779 4907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:18.100093 4907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:18.100212 4907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:18.100440 4907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:17:18.110217 4907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:33433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43351
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:18.111581 4907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:18.113089 4907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:18.126592 4913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:18.126861 4912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:18.127604 4915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:18.127791 4907 server_base.cc:1034] running on GCE node
I20260928 02:17:18.128548 4907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:18.129621 4907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:18.130849 4907 hybrid_clock.cc:648] HybridClock initialized: now 1790561838130776 us; error 63 us; skew 500 ppm
I20260928 02:17:18.131178 4907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:18 dist-test-slave-h37r krb5kdc[4666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561838, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:18.134599 4907 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:17:18.136148 4907 webserver.cc:460] Webserver started at http://127.4.140.194:32823/ using document root <none> and password file <none>
I20260928 02:17:18.136859 4907 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:18.136986 4907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:18.137318 4907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:18.140331 4907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "b4874676f9ad49048895b994a8c00f57"
format_stamp: "Formatted at 2026-09-28 02:17:18 on dist-test-slave-h37r"
server_key: "3c3fa9680da5f0287e2b2bf48c5dffa7"
server_key_iv: "61400473837f6bc44d9947dd61e92fa5"
server_key_version: "encryptionkey@0"
I20260928 02:17:18.141085 4907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "b4874676f9ad49048895b994a8c00f57"
format_stamp: "Formatted at 2026-09-28 02:17:18 on dist-test-slave-h37r"
server_key: "3c3fa9680da5f0287e2b2bf48c5dffa7"
server_key_iv: "61400473837f6bc44d9947dd61e92fa5"
server_key_version: "encryptionkey@0"
I20260928 02:17:18.146396 4907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260928 02:17:18.150226 4922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:18.151697 4907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 02:17:18.151881 4907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "b4874676f9ad49048895b994a8c00f57"
format_stamp: "Formatted at 2026-09-28 02:17:18 on dist-test-slave-h37r"
server_key: "3c3fa9680da5f0287e2b2bf48c5dffa7"
server_key_iv: "61400473837f6bc44d9947dd61e92fa5"
server_key_version: "encryptionkey@0"
I20260928 02:17:18.152135 4907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:18.179771 4907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:18.184432 4907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:18.184732 4907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:18.185731 4907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:18.187117 4907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:18.187196 4907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:18.187368 4907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:18.187417 4907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:18.221489 4907 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:33385
I20260928 02:17:18.221539 5034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:33385 every 8 connection(s)
I20260928 02:17:18.222797 4907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:17:18.224807 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 4907
I20260928 02:17:18.225039 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:17:18.225507 4659 external_mini_cluster.cc:1428] Setting key 16158342278fda02540101dea677d58d
Sep 28 02:17:18 dist-test-slave-h37r krb5kdc[4666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561838, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.140.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.195@KRBTEST.COM" created.
I20260928 02:17:18.243602 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:18.225699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:36873 (local address 127.4.140.254:43351)
0928 02:17:18.225912 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:18.225918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:18.227245 (+ 1327us) server_negotiation.cc:407] Connection header received
0928 02:17:18.229165 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:18.229174 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:18.229263 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:18.229379 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:18.230899 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:18.231603 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:18.233212 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:18.233624 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:18.233760 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:18.237440 (+ 3680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:18.237446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:18.237449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:18.237480 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:18.240255 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:18.241119 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:18.241129 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:18.241131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:18.241259 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:18.241793 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:18.241797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:18.241799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:18.242065 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:18.242189 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:18.242941 (+ 752us) server_negotiation.cc:299] Negotiation successful
0928 02:17:18.243107 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:17:18.244997 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:18.226226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43351 (local address 127.4.140.194:36873)
0928 02:17:18.227094 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:18.227145 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:18.228898 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:18.229586 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:18.229594 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:18.230132 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:18.230728 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:18.230745 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:18.231759 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:18.231764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:18.233061 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:18.233072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:18.233844 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:18.233851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:18.234148 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:18.234873 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:18.234908 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:18.237083 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:18.240407 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:18.240416 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:18.240434 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:18.240986 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:18.241374 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:18.241379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:18.241388 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:18.241624 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:18.242184 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:18.242190 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:18.242516 (+ 326us) client_negotiation.cc:769] Sending connection context
0928 02:17:18.242835 (+ 319us) client_negotiation.cc:240] Negotiation successful
0928 02:17:18.243342 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":370,"threads_started":1}
I20260928 02:17:18.246577 5035 heartbeater.cc:347] Connected to a master server at 127.4.140.254:43351
I20260928 02:17:18.246966 5035 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:18.247692 5035 heartbeater.cc:511] Master 127.4.140.254:43351 requested a full tablet report, sending...
I20260928 02:17:18.249501 4711 ts_manager.cc:194] Registered new tserver with Master: b4874676f9ad49048895b994a8c00f57 (127.4.140.194:33385)
I20260928 02:17:18.250329 4711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:36873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
I20260928 02:17:18.268133 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43351
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:33433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:18.587934 5040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:18.588235 5040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:18.588408 5040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:18.595677 5040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:18.595793 5040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:18.596009 5040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:17:18.605726 5040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:33433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43351
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:18.607096 5040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:18.608453 5040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:18.621742 5048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:18.621806 5046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:18.621961 5045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:18.622977 5040 server_base.cc:1034] running on GCE node
I20260928 02:17:18.623605 5040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:18.624651 5040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:18.625866 5040 hybrid_clock.cc:648] HybridClock initialized: now 1790561838625819 us; error 39 us; skew 500 ppm
I20260928 02:17:18.626206 5040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:18 dist-test-slave-h37r krb5kdc[4666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561838, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:18.629390 5040 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:17:18.630904 5040 webserver.cc:460] Webserver started at http://127.4.140.195:33765/ using document root <none> and password file <none>
I20260928 02:17:18.631603 5040 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:18.631747 5040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:18.632128 5040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:18.635203 5040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "a813fd7f0e6144eaa26e953e29890fd0"
format_stamp: "Formatted at 2026-09-28 02:17:18 on dist-test-slave-h37r"
server_key: "8cfffd56c3e9da96132581e14b260c54"
server_key_iv: "0a2a8c48c103f5325898856996c23d29"
server_key_version: "encryptionkey@0"
I20260928 02:17:18.636046 5040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "a813fd7f0e6144eaa26e953e29890fd0"
format_stamp: "Formatted at 2026-09-28 02:17:18 on dist-test-slave-h37r"
server_key: "8cfffd56c3e9da96132581e14b260c54"
server_key_iv: "0a2a8c48c103f5325898856996c23d29"
server_key_version: "encryptionkey@0"
I20260928 02:17:18.641281 5040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 02:17:18.645015 5055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:18.646549 5040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260928 02:17:18.646747 5040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "a813fd7f0e6144eaa26e953e29890fd0"
format_stamp: "Formatted at 2026-09-28 02:17:18 on dist-test-slave-h37r"
server_key: "8cfffd56c3e9da96132581e14b260c54"
server_key_iv: "0a2a8c48c103f5325898856996c23d29"
server_key_version: "encryptionkey@0"
I20260928 02:17:18.646972 5040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:18.662554 5040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:18.666909 5040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:18.667212 5040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:18.668346 5040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:18.669852 5040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:18.669951 5040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:18.670094 5040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:18.670168 5040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:18.703550 5040 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:41997
I20260928 02:17:18.703593 5167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:41997 every 8 connection(s)
I20260928 02:17:18.704934 5040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:17:18.715286 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 5040
I20260928 02:17:18.715487 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:17:18.716094 4659 external_mini_cluster.cc:1428] Setting key a6d5d77ce9c3f0bc390fabcb610c267e
Sep 28 02:17:18 dist-test-slave-h37r krb5kdc[4666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561838, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:18.725863 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:18.707601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:40671 (local address 127.4.140.254:43351)
0928 02:17:18.707779 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:18.707784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:18.709015 (+ 1231us) server_negotiation.cc:407] Connection header received
0928 02:17:18.710885 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:18.710890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:18.711012 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:18.711126 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:18.712737 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:18.713476 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:18.715033 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:18.715507 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:18.715645 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:18.720002 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:18.720008 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:18.720012 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:18.720056 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:18.722871 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:18.723746 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:18.723750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:18.723752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:18.723883 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:18.724334 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:18.724338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:18.724340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:18.724724 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:18.724849 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:18.725467 (+ 618us) server_negotiation.cc:299] Negotiation successful
0928 02:17:18.725628 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:17:18.726836 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:18.708150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43351 (local address 127.4.140.195:40671)
0928 02:17:18.708848 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:18.708890 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:18.710586 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:18.711374 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:18.711383 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:18.711980 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:18.712566 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:18.712583 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:18.713630 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:18.713636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:18.714874 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:18.714885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:18.715822 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:18.715830 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:18.716170 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:18.716909 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:18.716942 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:18.719692 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:18.723020 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:18.723028 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:18.723044 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:18.723610 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:18.724026 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:18.724029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:18.724031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:18.724201 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:18.724841 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:18.724846 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:18.725144 (+ 298us) client_negotiation.cc:769] Sending connection context
0928 02:17:18.725356 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 02:17:18.725744 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":270,"threads_started":1}
I20260928 02:17:18.728394 5168 heartbeater.cc:347] Connected to a master server at 127.4.140.254:43351
I20260928 02:17:18.728727 5168 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:18.729489 5168 heartbeater.cc:511] Master 127.4.140.254:43351 requested a full tablet report, sending...
I20260928 02:17:18.730985 4711 ts_manager.cc:194] Registered new tserver with Master: a813fd7f0e6144eaa26e953e29890fd0 (127.4.140.195:41997)
I20260928 02:17:18.731854 4711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:40671
I20260928 02:17:18.732452 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:17:18 dist-test-slave-h37r krb5kdc[4666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561838, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260928 02:17:18.756369 4901 heartbeater.cc:503] Master 127.4.140.254:43351 was elected leader, sending a full tablet report...
Sep 28 02:17:18 dist-test-slave-h37r krb5kdc[4666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561838, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:17:18.780429 5175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:18.764555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55022 (local address 127.4.140.193:36185)
0928 02:17:18.765436 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:18.765451 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:18.765870 (+ 419us) server_negotiation.cc:407] Connection header received
0928 02:17:18.766083 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:18.766090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:18.766305 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:18.766446 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:18.767958 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:18.769084 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:18.770337 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:18.770886 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:18.770982 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:18.774938 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:18.774949 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:18.774965 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:18.774996 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:18.777782 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:18.778432 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:18.778439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:18.778442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:18.778559 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:18.778926 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:18.778929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:18.778931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:18.779255 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:18.779486 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:18.779738 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 02:17:18.780023 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":412,"threads_started":1}
W20260928 02:17:18.781775 4816 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55022
Sep 28 02:17:18 dist-test-slave-h37r krb5kdc[4666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:17:18 dist-test-slave-h37r krb5kdc[4666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:17:18.825222 5175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:18.812037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55034 (local address 127.4.140.193:36185)
0928 02:17:18.812280 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:18.812285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:18.812469 (+ 184us) server_negotiation.cc:407] Connection header received
0928 02:17:18.812627 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:18.812631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:18.812734 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:18.812821 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:18.814151 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:18.814893 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:18.816215 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:18.816658 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:18.816781 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:18.820500 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:18.820505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:18.820508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:18.820540 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:18.822893 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:18.823511 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:18.823516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:18.823518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:18.823636 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:18.824043 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:18.824046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:18.824048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:18.824279 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:18.824441 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:18.824720 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 02:17:18.824888 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 02:17:18.826660 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 4773
I20260928 02:17:18.839146 5175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:18.827560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55050 (local address 127.4.140.193:36185)
0928 02:17:18.827773 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:18.827779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:18.828121 (+ 342us) server_negotiation.cc:407] Connection header received
0928 02:17:18.828303 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:18.828309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:18.828368 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:18.828460 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:18.829555 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:18.830246 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:18.831550 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:18.832005 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:18.832125 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:18.834243 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:18.834248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:18.834253 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:18.834289 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:18.836704 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:18.837305 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:18.837310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:18.837313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:18.837436 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:18.837778 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:18.837785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:18.837791 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:18.838032 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:18.838282 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:18.838594 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 02:17:18.838799 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:17:18.839838 4896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:18.923480 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 4773
I20260928 02:17:18.940131 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 4907
Sep 28 02:17:18 dist-test-slave-h37r krb5kdc[4666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.194@KRBTEST.COM
I20260928 02:17:18.955644 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:18.941047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55460 (local address 127.4.140.194:33385)
0928 02:17:18.941582 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:18.941589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:18.941620 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:17:18.941677 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:18.941684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:18.941903 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:18.942065 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:18.943132 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:18.944220 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:18.945370 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:18.945898 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:18.945996 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:18.949868 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:18.949879 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:18.949893 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:18.949924 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:18.952801 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:18.953449 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:18.953456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:18.953459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:18.953596 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:18.954028 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:18.954034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:18.954036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:18.954470 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:18.954726 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:18.955041 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 02:17:18.955335 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":291,"threads_started":1}
I20260928 02:17:18.957118 5030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:19.040040 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 4907
I20260928 02:17:19.062418 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 5040
Sep 28 02:17:19 dist-test-slave-h37r krb5kdc[4666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.195@KRBTEST.COM
I20260928 02:17:19.079259 5184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:19.063470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.4.140.195:41997)
0928 02:17:19.064173 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:19.064181 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:19.064211 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:17:19.064271 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:19.064279 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:19.064733 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:19.064875 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:19.066241 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:19.067376 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:19.068876 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:19.069497 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:19.069595 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:19.073683 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:19.073693 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:19.073707 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:19.073738 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:19.076617 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:19.077298 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:19.077305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:19.077307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:19.077429 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:19.077798 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:19.077801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:19.077803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:19.078112 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:19.078341 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:19.078740 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 02:17:19.078948 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":347,"threads_started":1}
I20260928 02:17:19.080591 5163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:19.163082 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 5040
I20260928 02:17:19.180039 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 4679
I20260928 02:17:19.181519 4741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:19.285415 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 4679
2026-09-28T02:17:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3893 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:17:19 dist-test-slave-h37r krb5kdc[5189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:17:19 dist-test-slave-h37r krb5kdc[5189](info): set up 2 sockets
Sep 28 02:17:19 dist-test-slave-h37r krb5kdc[5189](info): commencing operation
krb5kdc: starting...
W20260928 02:17:20.367830 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:17:20 dist-test-slave-h37r krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:17:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:17:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:17:20.546980 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:46323
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:36627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:46323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:20.879235 5202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:20.879606 5202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:20.879762 5202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:20.887108 5202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:17:20.887261 5202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:20.887367 5202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:20.887468 5202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:17:20.887553 5202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:17:20.897598 5202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:46323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:46323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:20.898945 5202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:20.900297 5202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:20.912886 5208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:20.912945 5210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:20.912863 5207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:20.913662 5202 server_base.cc:1034] running on GCE node
I20260928 02:17:20.914465 5202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:20.915974 5202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:20.917225 5202 hybrid_clock.cc:648] HybridClock initialized: now 1790561840917173 us; error 53 us; skew 500 ppm
I20260928 02:17:20.917616 5202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:20 dist-test-slave-h37r krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561840, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:20.920661 5202 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:17:20.922120 5202 webserver.cc:460] Webserver started at http://127.4.140.254:40635/ using document root <none> and password file <none>
I20260928 02:17:20.922840 5202 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:20.922981 5202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:20.923290 5202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:20.926245 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "9a3d44019b7442c39148c60ccf014aff"
format_stamp: "Formatted at 2026-09-28 02:17:20 on dist-test-slave-h37r"
server_key: "1e9d3d6c4c9502adb95489fc964a98ae"
server_key_iv: "37396335b4e726044691ae6c4176cc64"
server_key_version: "encryptionkey@0"
I20260928 02:17:20.927003 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "9a3d44019b7442c39148c60ccf014aff"
format_stamp: "Formatted at 2026-09-28 02:17:20 on dist-test-slave-h37r"
server_key: "1e9d3d6c4c9502adb95489fc964a98ae"
server_key_iv: "37396335b4e726044691ae6c4176cc64"
server_key_version: "encryptionkey@0"
I20260928 02:17:20.932216 5202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:17:20.935843 5217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:20.937505 5202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:17:20.937647 5202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "9a3d44019b7442c39148c60ccf014aff"
format_stamp: "Formatted at 2026-09-28 02:17:20 on dist-test-slave-h37r"
server_key: "1e9d3d6c4c9502adb95489fc964a98ae"
server_key_iv: "37396335b4e726044691ae6c4176cc64"
server_key_version: "encryptionkey@0"
I20260928 02:17:20.937820 5202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:20.954466 5202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:20.958568 5202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:20.958807 5202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:20.977020 5202 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:46323
I20260928 02:17:20.977041 5268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:46323 every 8 connection(s)
I20260928 02:17:20.978399 5202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:17:20.982113 5269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:20.984233 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 5202
I20260928 02:17:20.984473 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:17:20.985272 4659 external_mini_cluster.cc:1428] Setting key 34b7174666bf2887937ea3d6bc60b284
I20260928 02:17:20.993932 5269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff: Bootstrap starting.
I20260928 02:17:20.997231 5269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:20.998210 5269 log.cc:824] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:17:20 dist-test-slave-h37r krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:21.001596 5269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff: No bootstrap required, opened a new log
I20260928 02:17:21.008966 5272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:20.987376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53602 (local address 127.4.140.254:46323)
0928 02:17:20.988185 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:20.988197 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:20.988240 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:17:20.989875 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:20.989899 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:20.990297 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:20.990687 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:20.992266 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:20.993560 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:20.994900 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:20.995620 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:20.995832 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:21.000061 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:21.000072 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:21.000091 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:21.000131 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:21.004081 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:21.004814 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:21.004823 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:21.004832 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:21.005050 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:21.005483 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:21.005488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:21.005492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:21.006227 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:21.006518 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:21.006960 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 02:17:21.007487 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":327,"threads_started":1}
I20260928 02:17:21.012130 5269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3d44019b7442c39148c60ccf014aff" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 46323 } }
I20260928 02:17:21.012499 5269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:21.012575 5269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3d44019b7442c39148c60ccf014aff, State: Initialized, Role: FOLLOWER
I20260928 02:17:21.013130 5269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3d44019b7442c39148c60ccf014aff" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 46323 } }
I20260928 02:17:21.013327 5269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:17:21.013424 5269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:17:21.013542 5269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:21.015635 5269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3d44019b7442c39148c60ccf014aff" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 46323 } }
I20260928 02:17:21.016208 5269 leader_election.cc:302] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a3d44019b7442c39148c60ccf014aff; no voters:
I20260928 02:17:21.016819 5269 leader_election.cc:288] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:17:21.016978 5274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:21.017656 5274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff [term 1 LEADER]: Becoming Leader. State: Replica: 9a3d44019b7442c39148c60ccf014aff, State: Running, Role: LEADER
I20260928 02:17:21.018206 5274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3d44019b7442c39148c60ccf014aff" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 46323 } }
I20260928 02:17:21.018966 5269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:17:21.021062 5276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a3d44019b7442c39148c60ccf014aff. Latest consensus state: current_term: 1 leader_uuid: "9a3d44019b7442c39148c60ccf014aff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3d44019b7442c39148c60ccf014aff" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 46323 } } }
I20260928 02:17:21.021418 5276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:21.021203 5275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a3d44019b7442c39148c60ccf014aff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3d44019b7442c39148c60ccf014aff" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 46323 } } }
I20260928 02:17:21.021620 5275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:21.023865 5281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:17:21.028851 5281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:17:21.037710 5281 catalog_manager.cc:1347] Generated new cluster ID: dd1d7803a4f643209c6e33e9b1852791
I20260928 02:17:21.037828 5281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:17:21.070667 5281 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:17:21.072100 5281 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:17:21.083938 5281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a3d44019b7442c39148c60ccf014aff: Generated new TSK 0
I20260928 02:17:21.084807 5281 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
I20260928 02:17:21.129375 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:46323
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:36627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:21.470534 5295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:21.470852 5295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:21.470970 5295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:21.479339 5295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:21.479477 5295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:21.479605 5295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:17:21.490521 5295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:46323
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:21.491818 5295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:21.493209 5295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:21.507058 5301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:21.507059 5303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:21.506991 5300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:21.507995 5295 server_base.cc:1034] running on GCE node
I20260928 02:17:21.508723 5295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:21.509675 5295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:21.510872 5295 hybrid_clock.cc:648] HybridClock initialized: now 1790561841510790 us; error 77 us; skew 500 ppm
I20260928 02:17:21.511221 5295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:21 dist-test-slave-h37r krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561841, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:21.514417 5295 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:17:21.516350 5295 webserver.cc:460] Webserver started at http://127.4.140.193:34499/ using document root <none> and password file <none>
I20260928 02:17:21.517093 5295 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:21.517235 5295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:21.517561 5295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:21.521006 5295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "bd566ab2305943f385857fa3ac0fe3ba"
format_stamp: "Formatted at 2026-09-28 02:17:21 on dist-test-slave-h37r"
server_key: "f0f71a118f1b2bbfc2b46580cdf39a60"
server_key_iv: "902a73c50b56292eaf5594028cb20c73"
server_key_version: "encryptionkey@0"
I20260928 02:17:21.521996 5295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "bd566ab2305943f385857fa3ac0fe3ba"
format_stamp: "Formatted at 2026-09-28 02:17:21 on dist-test-slave-h37r"
server_key: "f0f71a118f1b2bbfc2b46580cdf39a60"
server_key_iv: "902a73c50b56292eaf5594028cb20c73"
server_key_version: "encryptionkey@0"
I20260928 02:17:21.527837 5295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:17:21.531749 5310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:21.533578 5295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:17:21.533761 5295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "bd566ab2305943f385857fa3ac0fe3ba"
format_stamp: "Formatted at 2026-09-28 02:17:21 on dist-test-slave-h37r"
server_key: "f0f71a118f1b2bbfc2b46580cdf39a60"
server_key_iv: "902a73c50b56292eaf5594028cb20c73"
server_key_version: "encryptionkey@0"
I20260928 02:17:21.533986 5295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:21.548611 5295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:21.552807 5295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:21.553115 5295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:21.554137 5295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:21.555482 5295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:21.555588 5295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:21.555742 5295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:21.555826 5295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:21.588534 5295 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:40495
I20260928 02:17:21.588570 5422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:40495 every 8 connection(s)
I20260928 02:17:21.589875 5295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:17:21.597265 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 5295
I20260928 02:17:21.597499 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:17:21.598111 4659 external_mini_cluster.cc:1428] Setting key dadd303ba5310195e89e4faae7d9b04a
Sep 28 02:17:21 dist-test-slave-h37r krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561841, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:21.610805 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:21.592507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:36693 (local address 127.4.140.254:46323)
0928 02:17:21.592901 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:21.592905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:21.594005 (+ 1100us) server_negotiation.cc:407] Connection header received
0928 02:17:21.595859 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:21.595865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:21.595979 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:21.596098 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:21.597800 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:21.598577 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:21.600520 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:21.600918 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:21.601052 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:21.604763 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:21.604771 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:21.604776 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:21.604817 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:21.607420 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:21.608389 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:21.608392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:21.608395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:21.608505 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:21.609010 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:21.609013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:21.609015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:21.609256 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:21.609361 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:21.610372 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0928 02:17:21.610563 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":194,"threads_started":1}
I20260928 02:17:21.612461 5426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:21.593127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:46323 (local address 127.4.140.193:36693)
0928 02:17:21.593820 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:21.593874 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:21.595578 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:21.596342 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:21.596352 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:21.596924 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:21.597622 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:21.597652 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:21.598737 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:21.598743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:21.600381 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:21.600393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:21.601158 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:21.601165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:21.601409 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:21.602100 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:21.602124 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:21.604526 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:21.607568 (+ 3042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:21.607579 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:21.607600 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:21.608276 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:21.608621 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:21.608628 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:21.608634 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:21.608900 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:21.609384 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:21.609395 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:21.609885 (+ 490us) client_negotiation.cc:769] Sending connection context
0928 02:17:21.610214 (+ 329us) client_negotiation.cc:240] Negotiation successful
0928 02:17:21.610829 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":263,"threads_started":1}
I20260928 02:17:21.614104 5423 heartbeater.cc:347] Connected to a master server at 127.4.140.254:46323
I20260928 02:17:21.614436 5423 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.140.194@KRBTEST.COM; defaulting to no policy
I20260928 02:17:21.615203 5423 heartbeater.cc:511] Master 127.4.140.254:46323 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.194@KRBTEST.COM" created.
I20260928 02:17:21.617383 5234 ts_manager.cc:194] Registered new tserver with Master: bd566ab2305943f385857fa3ac0fe3ba (127.4.140.193:40495)
I20260928 02:17:21.619623 5234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:36693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
I20260928 02:17:21.639545 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:46323
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:36627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:21.965076 5429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:21.965407 5429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:21.965581 5429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:21.972810 5429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:21.972965 5429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:21.973137 5429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:17:21.982980 5429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:46323
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:21.984318 5429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:21.985858 5429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:21.999074 5434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:21.999047 5437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:21.999161 5435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:21.999781 5429 server_base.cc:1034] running on GCE node
I20260928 02:17:22.000443 5429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:22.001329 5429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:22.002523 5429 hybrid_clock.cc:648] HybridClock initialized: now 1790561842002477 us; error 41 us; skew 500 ppm
I20260928 02:17:22.002797 5429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:22 dist-test-slave-h37r krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561842, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:22.006101 5429 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:17:22.007623 5429 webserver.cc:460] Webserver started at http://127.4.140.194:42173/ using document root <none> and password file <none>
I20260928 02:17:22.008400 5429 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:22.008526 5429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:22.008845 5429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:22.011715 5429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "c99110b9c87142cda346a92321ca10cd"
format_stamp: "Formatted at 2026-09-28 02:17:22 on dist-test-slave-h37r"
server_key: "bb29d8155ada5cca68c17c871eb97c6f"
server_key_iv: "7760a817a3d1c3aeed4fc6f34a9d2b46"
server_key_version: "encryptionkey@0"
I20260928 02:17:22.012586 5429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "c99110b9c87142cda346a92321ca10cd"
format_stamp: "Formatted at 2026-09-28 02:17:22 on dist-test-slave-h37r"
server_key: "bb29d8155ada5cca68c17c871eb97c6f"
server_key_iv: "7760a817a3d1c3aeed4fc6f34a9d2b46"
server_key_version: "encryptionkey@0"
I20260928 02:17:22.017953 5429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:17:22.021562 5444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:22.022940 5429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:17:22.023138 5429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "c99110b9c87142cda346a92321ca10cd"
format_stamp: "Formatted at 2026-09-28 02:17:22 on dist-test-slave-h37r"
server_key: "bb29d8155ada5cca68c17c871eb97c6f"
server_key_iv: "7760a817a3d1c3aeed4fc6f34a9d2b46"
server_key_version: "encryptionkey@0"
I20260928 02:17:22.023353 5429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:22.056425 5429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:22.060309 5429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:22.060599 5429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:22.061558 5429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:22.062848 5429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:22.062952 5429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:22.063112 5429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:22.063184 5429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:22.096947 5429 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:40011
I20260928 02:17:22.097039 5556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:40011 every 8 connection(s)
I20260928 02:17:22.098271 5429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:17:22.107523 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 5429
I20260928 02:17:22.107751 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:17:22.108242 4659 external_mini_cluster.cc:1428] Setting key 9103f23f70f076e042eb56ad34935645
Sep 28 02:17:22 dist-test-slave-h37r krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561842, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:22.119935 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:22.101100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:49227 (local address 127.4.140.254:46323)
0928 02:17:22.101308 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:22.101314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:22.102549 (+ 1235us) server_negotiation.cc:407] Connection header received
0928 02:17:22.104491 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:22.104496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:22.104591 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:22.104710 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:22.106144 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.106872 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.108708 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.109285 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.109408 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:22.113635 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:22.113640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:22.113644 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:22.113673 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:22.116578 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:22.117344 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:22.117353 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:22.117359 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:22.117511 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:22.117922 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:22.117930 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:22.117936 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:22.118306 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:22.118461 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:22.119201 (+ 740us) server_negotiation.cc:299] Negotiation successful
0928 02:17:22.119470 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:17:22.120935 5559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:22.101576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:46323 (local address 127.4.140.194:49227)
0928 02:17:22.102363 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:22.102408 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:22.104212 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:22.104909 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:22.104917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:22.105413 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:22.105965 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:22.105983 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.107015 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.107019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:22.108541 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:22.108554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.109526 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.109533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:22.109794 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:22.110529 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:22.110552 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:22.113385 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:22.116715 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:22.116722 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:22.116735 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:22.117163 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:22.117620 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:22.117623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:22.117625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:22.117805 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:22.118456 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:22.118462 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:22.118807 (+ 345us) client_negotiation.cc:769] Sending connection context
0928 02:17:22.119099 (+ 292us) client_negotiation.cc:240] Negotiation successful
0928 02:17:22.119620 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":347,"threads_started":1}
I20260928 02:17:22.122473 5557 heartbeater.cc:347] Connected to a master server at 127.4.140.254:46323
I20260928 02:17:22.122797 5557 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:22.123548 5557 heartbeater.cc:511] Master 127.4.140.254:46323 requested a full tablet report, sending...
I20260928 02:17:22.125296 5234 ts_manager.cc:194] Registered new tserver with Master: c99110b9c87142cda346a92321ca10cd (127.4.140.194:40011)
WARNING: no policy specified for kudu/127.4.140.195@KRBTEST.COM; defaulting to no policy
I20260928 02:17:22.126276 5234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:49227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
I20260928 02:17:22.149497 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:46323
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:36627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:22.467280 5562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:22.467574 5562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:22.467671 5562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:22.475100 5562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:22.475214 5562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:22.475335 5562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:17:22.485262 5562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:46323
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:22.486541 5562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:22.487830 5562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:22.501116 5567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:22.501055 5568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:22.501145 5570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:22.502202 5562 server_base.cc:1034] running on GCE node
I20260928 02:17:22.502945 5562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:22.503988 5562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:22.505215 5562 hybrid_clock.cc:648] HybridClock initialized: now 1790561842505171 us; error 56 us; skew 500 ppm
I20260928 02:17:22.505548 5562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:22 dist-test-slave-h37r krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561842, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:22.508680 5562 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:17:22.510257 5562 webserver.cc:460] Webserver started at http://127.4.140.195:36987/ using document root <none> and password file <none>
I20260928 02:17:22.510922 5562 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:22.511018 5562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:22.511265 5562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:22.514174 5562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "c7136ece40334b6fbc333e39fb802c57"
format_stamp: "Formatted at 2026-09-28 02:17:22 on dist-test-slave-h37r"
server_key: "4a1d0067f83a9587f17a1cd878dc95a1"
server_key_iv: "071a080ebda39ea3f9412d4f2e8fc6d7"
server_key_version: "encryptionkey@0"
I20260928 02:17:22.514902 5562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "c7136ece40334b6fbc333e39fb802c57"
format_stamp: "Formatted at 2026-09-28 02:17:22 on dist-test-slave-h37r"
server_key: "4a1d0067f83a9587f17a1cd878dc95a1"
server_key_iv: "071a080ebda39ea3f9412d4f2e8fc6d7"
server_key_version: "encryptionkey@0"
I20260928 02:17:22.520376 5562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:17:22.523865 5577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:22.525339 5562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:17:22.525512 5562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "c7136ece40334b6fbc333e39fb802c57"
format_stamp: "Formatted at 2026-09-28 02:17:22 on dist-test-slave-h37r"
server_key: "4a1d0067f83a9587f17a1cd878dc95a1"
server_key_iv: "071a080ebda39ea3f9412d4f2e8fc6d7"
server_key_version: "encryptionkey@0"
I20260928 02:17:22.525693 5562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:22.551957 5562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:22.556066 5562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:22.556303 5562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:22.557235 5562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:22.558530 5562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:22.558599 5562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:22.558679 5562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:22.558722 5562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:22.593317 5562 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:37631
I20260928 02:17:22.593334 5689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:37631 every 8 connection(s)
I20260928 02:17:22.594615 5562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:17:22.596750 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 5562
I20260928 02:17:22.597002 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:17:22.597470 4659 external_mini_cluster.cc:1428] Setting key 60372a4dd210bfaddb5036f252f6bf8b
Sep 28 02:17:22 dist-test-slave-h37r krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561842, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:22.615260 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:22.597596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:49131 (local address 127.4.140.254:46323)
0928 02:17:22.597784 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:22.597790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:22.599240 (+ 1450us) server_negotiation.cc:407] Connection header received
0928 02:17:22.601165 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:22.601169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:22.601234 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:22.601373 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:22.602874 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.603625 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.605190 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.605602 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.605701 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:22.609497 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:22.609502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:22.609505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:22.609537 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:22.612384 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:22.613191 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:22.613194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:22.613197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:22.613326 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:22.613686 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:22.613689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:22.613691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:22.613946 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:22.614058 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:22.614789 (+ 731us) server_negotiation.cc:299] Negotiation successful
0928 02:17:22.614979 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:17:22.616276 5692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:22.598236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:46323 (local address 127.4.140.195:49131)
0928 02:17:22.599088 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:22.599139 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:22.600918 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:22.601617 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:22.601625 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:22.602148 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:22.602712 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:22.602728 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.603751 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.603755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:22.605055 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:22.605066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.605798 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.605803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:22.605999 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:22.606623 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:22.606646 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:22.609256 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:22.612521 (+ 3265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:22.612528 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:22.612541 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:22.613070 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:22.613419 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:22.613421 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:22.613424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:22.613593 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:22.614066 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:22.614072 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:22.614384 (+ 312us) client_negotiation.cc:769] Sending connection context
0928 02:17:22.614612 (+ 228us) client_negotiation.cc:240] Negotiation successful
0928 02:17:22.615031 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":348,"threads_started":1}
I20260928 02:17:22.617812 5690 heartbeater.cc:347] Connected to a master server at 127.4.140.254:46323
I20260928 02:17:22.618085 5690 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:22.618767 5690 heartbeater.cc:511] Master 127.4.140.254:46323 requested a full tablet report, sending...
I20260928 02:17:22.620322 5234 ts_manager.cc:194] Registered new tserver with Master: c7136ece40334b6fbc333e39fb802c57 (127.4.140.195:37631)
I20260928 02:17:22.621345 5234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:49131
I20260928 02:17:22.623008 5423 heartbeater.cc:503] Master 127.4.140.254:46323 was elected leader, sending a full tablet report...
I20260928 02:17:22.625423 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:17:22.645427 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:22.632069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.4.140.254:46323)
0928 02:17:22.632373 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:22.632381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:22.632691 (+ 310us) server_negotiation.cc:407] Connection header received
0928 02:17:22.632909 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:22.632914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:22.632986 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:22.633094 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:22.634523 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.635286 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.636683 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.637125 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.637278 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:22.639728 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:22.639739 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:22.639745 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:22.639777 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:22.642718 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:22.643396 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:22.643403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:22.643406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:22.643538 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:22.643954 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:22.643960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:22.643964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:22.644295 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:22.644565 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:22.644886 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 02:17:22.645097 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:17:22.652205 5234 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:17:22.658914 5234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:17:22.693670 5704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:22.671084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:40011 (local address 127.0.0.1:51906)
0928 02:17:22.682802 (+ 11718us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:22.682853 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:22.683066 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:22.684490 (+ 1424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:22.684502 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:22.684548 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:22.685090 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:22.685108 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.687794 (+ 2686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.687802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:22.690505 (+ 2703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:22.690525 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.692474 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.692480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:22.692763 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:22.692833 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 02:17:22.692985 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 02:17:22.693206 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11434,"mutex_wait_us":88,"thread_start_us":12703,"threads_started":1}
I20260928 02:17:22.693688 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:22.671245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51906 (local address 127.4.140.194:40011)
0928 02:17:22.683649 (+ 12404us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:22.683658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:22.683688 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:17:22.683813 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:22.683822 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:22.684170 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:22.685357 (+ 1187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:22.685672 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.687614 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.690704 (+ 3090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.692256 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.692402 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:22.692608 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:22.693189 (+ 581us) server_negotiation.cc:299] Negotiation successful
0928 02:17:22.693379 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12234,"thread_start_us":12134,"threads_started":1}
I20260928 02:17:22.693825 5702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:22.671259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:40495 (local address 127.0.0.1:57896)
0928 02:17:22.684091 (+ 12832us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:22.684109 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:22.684243 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:22.685399 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:22.685405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:22.685441 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:22.685973 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:22.685990 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.688145 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.688150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:22.691057 (+ 2907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:22.691070 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.692924 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.692931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:22.693126 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:22.693158 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:17:22.693251 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 02:17:22.693385 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12706,"mutex_wait_us":2,"thread_start_us":11116,"threads_started":1}
I20260928 02:17:22.694523 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:22.671778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.4.140.193:40495)
0928 02:17:22.682319 (+ 10541us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:22.682335 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:22.684410 (+ 2075us) server_negotiation.cc:407] Connection header received
0928 02:17:22.684543 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:22.684553 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:22.684848 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:22.685095 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:22.686207 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.687989 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.691251 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.692693 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.692848 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:22.693079 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:22.694045 (+ 966us) server_negotiation.cc:299] Negotiation successful
0928 02:17:22.694227 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10272,"thread_start_us":10076,"threads_started":1}
I20260928 02:17:22.696586 5707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:22.684162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:37631 (local address 127.0.0.1:49832)
0928 02:17:22.687827 (+ 3665us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:22.687850 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:22.688018 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:22.688775 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:22.688780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:22.688809 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:22.689203 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:22.689215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.690704 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.690712 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:22.692872 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:22.692883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.695769 (+ 2886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.695776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:22.695966 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:22.695996 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 02:17:22.696082 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:17:22.696211 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3506,"mutex_wait_us":54,"thread_start_us":3358,"threads_started":1}
I20260928 02:17:22.697304 5706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:22.683725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.4.140.195:37631)
0928 02:17:22.687490 (+ 3765us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:22.687499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:22.688152 (+ 653us) server_negotiation.cc:407] Connection header received
0928 02:17:22.688264 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:22.688272 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:22.688511 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:22.688701 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:22.689448 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.690510 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.693049 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.695268 (+ 2219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.695396 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:22.695616 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:22.696927 (+ 1311us) server_negotiation.cc:299] Negotiation successful
0928 02:17:22.697058 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3577,"thread_start_us":3434,"threads_started":1}
I20260928 02:17:22.698218 5358 tablet_service.cc:1467] Processing CreateTablet for tablet df27c9ff905f4845b98593a295ba3229 (DEFAULT_TABLE table=test-table [id=7ab053c5e6974234aa1a29d8d11e8b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:22.698218 5492 tablet_service.cc:1467] Processing CreateTablet for tablet df27c9ff905f4845b98593a295ba3229 (DEFAULT_TABLE table=test-table [id=7ab053c5e6974234aa1a29d8d11e8b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:22.699723 5358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df27c9ff905f4845b98593a295ba3229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:22.699738 5492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df27c9ff905f4845b98593a295ba3229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:22.700577 5625 tablet_service.cc:1467] Processing CreateTablet for tablet df27c9ff905f4845b98593a295ba3229 (DEFAULT_TABLE table=test-table [id=7ab053c5e6974234aa1a29d8d11e8b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:22.701874 5625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df27c9ff905f4845b98593a295ba3229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:22.714536 5709 tablet_bootstrap.cc:492] T df27c9ff905f4845b98593a295ba3229 P bd566ab2305943f385857fa3ac0fe3ba: Bootstrap starting.
I20260928 02:17:22.713783 5708 tablet_bootstrap.cc:492] T df27c9ff905f4845b98593a295ba3229 P c7136ece40334b6fbc333e39fb802c57: Bootstrap starting.
I20260928 02:17:22.716233 5710 tablet_bootstrap.cc:492] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd: Bootstrap starting.
I20260928 02:17:22.718432 5708 tablet_bootstrap.cc:654] T df27c9ff905f4845b98593a295ba3229 P c7136ece40334b6fbc333e39fb802c57: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:22.718432 5709 tablet_bootstrap.cc:654] T df27c9ff905f4845b98593a295ba3229 P bd566ab2305943f385857fa3ac0fe3ba: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:22.719332 5710 tablet_bootstrap.cc:654] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:22.719813 5709 log.cc:824] T df27c9ff905f4845b98593a295ba3229 P bd566ab2305943f385857fa3ac0fe3ba: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:22.720293 5708 log.cc:824] T df27c9ff905f4845b98593a295ba3229 P c7136ece40334b6fbc333e39fb802c57: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:22.720534 5710 log.cc:824] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:22.723126 5710 tablet_bootstrap.cc:492] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd: No bootstrap required, opened a new log
I20260928 02:17:22.723181 5709 tablet_bootstrap.cc:492] T df27c9ff905f4845b98593a295ba3229 P bd566ab2305943f385857fa3ac0fe3ba: No bootstrap required, opened a new log
I20260928 02:17:22.723207 5708 tablet_bootstrap.cc:492] T df27c9ff905f4845b98593a295ba3229 P c7136ece40334b6fbc333e39fb802c57: No bootstrap required, opened a new log
I20260928 02:17:22.723418 5710 ts_tablet_manager.cc:1397] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 02:17:22.723441 5709 ts_tablet_manager.cc:1397] T df27c9ff905f4845b98593a295ba3229 P bd566ab2305943f385857fa3ac0fe3ba: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 02:17:22.723450 5708 ts_tablet_manager.cc:1397] T df27c9ff905f4845b98593a295ba3229 P c7136ece40334b6fbc333e39fb802c57: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 02:17:22.731962 5710 raft_consensus.cc:348] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c99110b9c87142cda346a92321ca10cd" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 40011 } } peers { permanent_uuid: "bd566ab2305943f385857fa3ac0fe3ba" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 40495 } } peers { permanent_uuid: "c7136ece40334b6fbc333e39fb802c57" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 37631 } }
I20260928 02:17:22.732395 5710 raft_consensus.cc:374] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:22.732494 5710 raft_consensus.cc:729] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c99110b9c87142cda346a92321ca10cd, State: Initialized, Role: FOLLOWER
I20260928 02:17:22.733148 5710 consensus_queue.cc:260] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c99110b9c87142cda346a92321ca10cd" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 40011 } } peers { permanent_uuid: "bd566ab2305943f385857fa3ac0fe3ba" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 40495 } } peers { permanent_uuid: "c7136ece40334b6fbc333e39fb802c57" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 37631 } }
I20260928 02:17:22.734315 5557 heartbeater.cc:503] Master 127.4.140.254:46323 was elected leader, sending a full tablet report...
I20260928 02:17:22.734193 5709 raft_consensus.cc:348] T df27c9ff905f4845b98593a295ba3229 P bd566ab2305943f385857fa3ac0fe3ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c99110b9c87142cda346a92321ca10cd" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 40011 } } peers { permanent_uuid: "bd566ab2305943f385857fa3ac0fe3ba" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 40495 } } peers { permanent_uuid: "c7136ece40334b6fbc333e39fb802c57" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 37631 } }
I20260928 02:17:22.734627 5709 raft_consensus.cc:374] T df27c9ff905f4845b98593a295ba3229 P bd566ab2305943f385857fa3ac0fe3ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:22.734714 5709 raft_consensus.cc:729] T df27c9ff905f4845b98593a295ba3229 P bd566ab2305943f385857fa3ac0fe3ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd566ab2305943f385857fa3ac0fe3ba, State: Initialized, Role: FOLLOWER
I20260928 02:17:22.734601 5708 raft_consensus.cc:348] T df27c9ff905f4845b98593a295ba3229 P c7136ece40334b6fbc333e39fb802c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c99110b9c87142cda346a92321ca10cd" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 40011 } } peers { permanent_uuid: "bd566ab2305943f385857fa3ac0fe3ba" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 40495 } } peers { permanent_uuid: "c7136ece40334b6fbc333e39fb802c57" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 37631 } }
I20260928 02:17:22.735013 5708 raft_consensus.cc:374] T df27c9ff905f4845b98593a295ba3229 P c7136ece40334b6fbc333e39fb802c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:22.735038 5710 ts_tablet_manager.cc:1428] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260928 02:17:22.735107 5708 raft_consensus.cc:729] T df27c9ff905f4845b98593a295ba3229 P c7136ece40334b6fbc333e39fb802c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7136ece40334b6fbc333e39fb802c57, State: Initialized, Role: FOLLOWER
I20260928 02:17:22.735420 5709 consensus_queue.cc:260] T df27c9ff905f4845b98593a295ba3229 P bd566ab2305943f385857fa3ac0fe3ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c99110b9c87142cda346a92321ca10cd" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 40011 } } peers { permanent_uuid: "bd566ab2305943f385857fa3ac0fe3ba" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 40495 } } peers { permanent_uuid: "c7136ece40334b6fbc333e39fb802c57" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 37631 } }
I20260928 02:17:22.737376 5709 ts_tablet_manager.cc:1428] T df27c9ff905f4845b98593a295ba3229 P bd566ab2305943f385857fa3ac0fe3ba: Time spent starting tablet: real 0.014s user 0.014s sys 0.002s
I20260928 02:17:22.738879 5708 consensus_queue.cc:260] T df27c9ff905f4845b98593a295ba3229 P c7136ece40334b6fbc333e39fb802c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c99110b9c87142cda346a92321ca10cd" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 40011 } } peers { permanent_uuid: "bd566ab2305943f385857fa3ac0fe3ba" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 40495 } } peers { permanent_uuid: "c7136ece40334b6fbc333e39fb802c57" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 37631 } }
I20260928 02:17:22.745258 5708 ts_tablet_manager.cc:1428] T df27c9ff905f4845b98593a295ba3229 P c7136ece40334b6fbc333e39fb802c57: Time spent starting tablet: real 0.022s user 0.012s sys 0.007s
I20260928 02:17:22.749198 5690 heartbeater.cc:503] Master 127.4.140.254:46323 was elected leader, sending a full tablet report...
I20260928 02:17:22.786604 5714 raft_consensus.cc:482] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:17:22.786994 5714 raft_consensus.cc:504] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c99110b9c87142cda346a92321ca10cd" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 40011 } } peers { permanent_uuid: "bd566ab2305943f385857fa3ac0fe3ba" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 40495 } } peers { permanent_uuid: "c7136ece40334b6fbc333e39fb802c57" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 37631 } }
I20260928 02:17:22.788928 5714 leader_election.cc:288] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd566ab2305943f385857fa3ac0fe3ba (127.4.140.193:40495), c7136ece40334b6fbc333e39fb802c57 (127.4.140.195:37631)
I20260928 02:17:22.797997 5706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:22.790490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:46727 (local address 127.4.140.195:37631)
0928 02:17:22.790704 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:22.790711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:22.791103 (+ 392us) server_negotiation.cc:407] Connection header received
0928 02:17:22.791355 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:22.791361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:22.791429 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:22.791484 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:22.792284 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.793504 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.795941 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.796813 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.797023 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:22.797117 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:22.797529 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 02:17:22.797687 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:17:22.798080 5718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:22.790341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:37631 (local address 127.4.140.194:46727)
0928 02:17:22.790993 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:22.791010 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:22.791206 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:22.791659 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:22.791665 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:22.791691 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:22.792095 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:22.792109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.793667 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.793673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:22.795732 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:22.795743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.797052 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.797059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:22.797304 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:22.797373 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 02:17:22.797480 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:17:22.797630 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":3,"thread_start_us":305,"threads_started":1}
I20260928 02:17:22.798144 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:22.789407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:49367 (local address 127.4.140.193:40495)
0928 02:17:22.789632 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:22.789638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:22.790299 (+ 661us) server_negotiation.cc:407] Connection header received
0928 02:17:22.790705 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:22.790711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:22.790796 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:22.790873 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:22.791721 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.792733 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.795416 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.796937 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.797139 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:22.797227 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:22.797750 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 02:17:22.797899 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:17:22.798897 5717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:22.789373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:40495 (local address 127.4.140.194:49367)
0928 02:17:22.790192 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:22.790216 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:22.790458 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:22.791092 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:22.791097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:22.791122 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:22.791533 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:22.791549 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.792866 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.792874 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:22.795228 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:22.795242 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.797301 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.797307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:22.797568 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:22.797600 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:17:22.798332 (+ 732us) client_negotiation.cc:240] Negotiation successful
0928 02:17:22.798479 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":54,"thread_start_us":442,"threads_started":1}
I20260928 02:17:22.799342 5645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df27c9ff905f4845b98593a295ba3229" candidate_uuid: "c99110b9c87142cda346a92321ca10cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7136ece40334b6fbc333e39fb802c57" is_pre_election: true
I20260928 02:17:22.799998 5645 raft_consensus.cc:2454] T df27c9ff905f4845b98593a295ba3229 P c7136ece40334b6fbc333e39fb802c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c99110b9c87142cda346a92321ca10cd in term 0.
I20260928 02:17:22.800031 5378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df27c9ff905f4845b98593a295ba3229" candidate_uuid: "c99110b9c87142cda346a92321ca10cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd566ab2305943f385857fa3ac0fe3ba" is_pre_election: true
I20260928 02:17:22.800634 5378 raft_consensus.cc:2454] T df27c9ff905f4845b98593a295ba3229 P bd566ab2305943f385857fa3ac0fe3ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c99110b9c87142cda346a92321ca10cd in term 0.
I20260928 02:17:22.800815 5446 leader_election.cc:302] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7136ece40334b6fbc333e39fb802c57, c99110b9c87142cda346a92321ca10cd; no voters:
I20260928 02:17:22.801388 5714 raft_consensus.cc:2781] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:17:22.801553 5714 raft_consensus.cc:482] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:17:22.801712 5714 raft_consensus.cc:3037] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:22.804162 5714 raft_consensus.cc:504] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c99110b9c87142cda346a92321ca10cd" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 40011 } } peers { permanent_uuid: "bd566ab2305943f385857fa3ac0fe3ba" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 40495 } } peers { permanent_uuid: "c7136ece40334b6fbc333e39fb802c57" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 37631 } }
I20260928 02:17:22.804798 5714 leader_election.cc:288] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd [CANDIDATE]: Term 1 election: Requested vote from peers bd566ab2305943f385857fa3ac0fe3ba (127.4.140.193:40495), c7136ece40334b6fbc333e39fb802c57 (127.4.140.195:37631)
I20260928 02:17:22.805289 5378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df27c9ff905f4845b98593a295ba3229" candidate_uuid: "c99110b9c87142cda346a92321ca10cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd566ab2305943f385857fa3ac0fe3ba"
I20260928 02:17:22.805343 5645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df27c9ff905f4845b98593a295ba3229" candidate_uuid: "c99110b9c87142cda346a92321ca10cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7136ece40334b6fbc333e39fb802c57"
I20260928 02:17:22.805543 5378 raft_consensus.cc:3037] T df27c9ff905f4845b98593a295ba3229 P bd566ab2305943f385857fa3ac0fe3ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:22.805568 5645 raft_consensus.cc:3037] T df27c9ff905f4845b98593a295ba3229 P c7136ece40334b6fbc333e39fb802c57 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:22.808005 5645 raft_consensus.cc:2454] T df27c9ff905f4845b98593a295ba3229 P c7136ece40334b6fbc333e39fb802c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c99110b9c87142cda346a92321ca10cd in term 1.
I20260928 02:17:22.808461 5446 leader_election.cc:302] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7136ece40334b6fbc333e39fb802c57, c99110b9c87142cda346a92321ca10cd; no voters:
I20260928 02:17:22.808773 5714 raft_consensus.cc:2781] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:22.809146 5378 raft_consensus.cc:2454] T df27c9ff905f4845b98593a295ba3229 P bd566ab2305943f385857fa3ac0fe3ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c99110b9c87142cda346a92321ca10cd in term 1.
I20260928 02:17:22.809432 5714 raft_consensus.cc:686] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd [term 1 LEADER]: Becoming Leader. State: Replica: c99110b9c87142cda346a92321ca10cd, State: Running, Role: LEADER
I20260928 02:17:22.809978 5714 consensus_queue.cc:237] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c99110b9c87142cda346a92321ca10cd" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 40011 } } peers { permanent_uuid: "bd566ab2305943f385857fa3ac0fe3ba" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 40495 } } peers { permanent_uuid: "c7136ece40334b6fbc333e39fb802c57" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 37631 } }
I20260928 02:17:22.818054 5232 catalog_manager.cc:5495] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd reported cstate change: term changed from 0 to 1, leader changed from <none> to c99110b9c87142cda346a92321ca10cd (127.4.140.194). New cstate: current_term: 1 leader_uuid: "c99110b9c87142cda346a92321ca10cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c99110b9c87142cda346a92321ca10cd" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 40011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd566ab2305943f385857fa3ac0fe3ba" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7136ece40334b6fbc333e39fb802c57" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 37631 } health_report { overall_health: UNKNOWN } } }
W20260928 02:17:22.842372 5424 tablet.cc:2369] T df27c9ff905f4845b98593a295ba3229 P bd566ab2305943f385857fa3ac0fe3ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:17:22.846240 5691 tablet.cc:2369] T df27c9ff905f4845b98593a295ba3229 P c7136ece40334b6fbc333e39fb802c57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:17:22.850464 5558 tablet.cc:2369] T df27c9ff905f4845b98593a295ba3229 P c99110b9c87142cda346a92321ca10cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:17:22 dist-test-slave-h37r krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:17:22.866904 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:22.852364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57908 (local address 127.4.140.193:40495)
0928 02:17:22.852578 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:22.852588 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:22.853080 (+ 492us) server_negotiation.cc:407] Connection header received
0928 02:17:22.853224 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:22.853228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:22.853292 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:22.853374 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:22.854838 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.855567 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.856876 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.857266 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.857380 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:22.861304 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:22.861319 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:22.861335 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:22.861369 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:22.864269 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:22.864892 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:22.864899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:22.864901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:22.865018 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:22.865456 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:22.865459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:22.865461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:22.865807 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:22.866023 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:22.866425 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 02:17:22.866586 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 02:17:22 dist-test-slave-h37r krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561842, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:17:22 dist-test-slave-h37r krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561842, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:17:22.914916 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:22.900456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57912 (local address 127.4.140.193:40495)
0928 02:17:22.900691 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:22.900699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:22.900955 (+ 256us) server_negotiation.cc:407] Connection header received
0928 02:17:22.901103 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:22.901108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:22.901173 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:22.901262 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:22.902661 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.903455 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.904876 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.905301 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.905408 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:22.909667 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:22.909673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:22.909677 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:22.909722 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:22.912499 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:22.913110 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:22.913114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:22.913117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:22.913299 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:22.913639 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:22.913642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:22.913645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:22.913929 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:22.914141 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:22.914364 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 02:17:22.914535 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 02:17:22 dist-test-slave-h37r krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:17:22 dist-test-slave-h37r krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:17:22.959353 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:22.946013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57914 (local address 127.4.140.193:40495)
0928 02:17:22.946255 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:22.946262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:22.946522 (+ 260us) server_negotiation.cc:407] Connection header received
0928 02:17:22.946661 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:22.946665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:22.946729 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:22.946818 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:22.948222 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.948902 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.950077 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.950498 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.950592 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:22.954461 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:22.954466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:22.954469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:22.954504 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:22.957050 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:22.957620 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:22.957624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:22.957626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:22.957741 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:22.958043 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:22.958046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:22.958048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:22.958373 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:22.958546 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:22.958786 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 02:17:22.958953 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 02:17:22.961155 5338 tablet_service.cc:2908] Rejecting scan request for tablet df27c9ff905f4845b98593a295ba3229: Uninitialized: clean time has not yet been initialized
I20260928 02:17:22.964176 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 5295
I20260928 02:17:22.976506 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:22.965107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57922 (local address 127.4.140.193:40495)
0928 02:17:22.965313 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:22.965319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:22.965548 (+ 229us) server_negotiation.cc:407] Connection header received
0928 02:17:22.965675 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:22.965679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:22.965757 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:22.965843 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:22.966977 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.967684 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.969068 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:22.969482 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:22.969594 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:22.971635 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:22.971640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:22.971643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:22.971671 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:22.974276 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:22.974855 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:22.974858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:22.974861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:22.974979 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:22.975264 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:22.975267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:22.975269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:22.975521 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:22.975739 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:22.975973 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 02:17:22.976145 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:17:22.977252 5418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:23.072440 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 5295
I20260928 02:17:23.090106 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 5429
Sep 28 02:17:23 dist-test-slave-h37r krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.194@KRBTEST.COM
I20260928 02:17:23.107234 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:23.091051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51918 (local address 127.4.140.194:40011)
0928 02:17:23.091271 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:23.091279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:23.091299 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:17:23.091405 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:23.091414 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:23.091474 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:23.091583 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:23.092839 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:23.093547 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:23.094835 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:23.095263 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:23.095445 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:23.100576 (+ 5131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:23.100587 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:23.100599 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:23.100635 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:23.104393 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:23.105042 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:23.105052 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:23.105056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:23.105230 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:23.105624 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:23.105629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:23.105633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:23.106036 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:23.106334 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:23.106613 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 02:17:23.106902 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:17:23.108057 5552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:23.198220 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 5429
I20260928 02:17:23.215538 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 5562
Sep 28 02:17:23 dist-test-slave-h37r krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.195@KRBTEST.COM
I20260928 02:17:23.231024 5706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:23.216314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49836 (local address 127.4.140.195:37631)
0928 02:17:23.216481 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:23.216485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:23.216504 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:17:23.216652 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:23.216656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:23.216718 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:23.216807 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:23.218047 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:23.218773 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:23.220221 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:23.220617 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:23.220752 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:23.225330 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:23.225346 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:23.225363 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:23.225398 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:23.228539 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:23.229198 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:23.229206 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:23.229208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:23.229374 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:23.229746 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:23.229751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:23.229753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:23.230120 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:23.230357 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:23.230554 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 02:17:23.230785 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:17:23.231637 5685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:23.324118 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 5562
I20260928 02:17:23.341399 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 5202
I20260928 02:17:23.342789 5264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:23.453419 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 5202
2026-09-28T02:17:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4169 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:17:25 dist-test-slave-h37r krb5kdc[5738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:17:25 dist-test-slave-h37r krb5kdc[5738](info): set up 2 sockets
Sep 28 02:17:25 dist-test-slave-h37r krb5kdc[5738](info): commencing operation
krb5kdc: starting...
W20260928 02:17:26.580741 4659 mini_kdc.cc:121] Time spent starting KDC: real 3.083s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 02:17:26.834517 4659 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.254s user 0.000s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260928 02:17:26.962284 4659 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.128s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 02:17:27.173524 4659 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.211s user 0.001s sys 0.005s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/test-user.keytab.
W20260928 02:17:27.279987 4659 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.106s user 0.000s sys 0.005s
Sep 28 02:17:27 dist-test-slave-h37r krb5kdc[5738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:17:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:17:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:17:27.363708 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:43303
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:36101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:43303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:27.705211 5751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:27.705587 5751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:27.705745 5751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:27.713032 5751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:17:27.713182 5751 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:17:27.713294 5751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:27.713402 5751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:27.713486 5751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:17:27.713577 5751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:17:27.723431 5751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:43303
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:43303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:27.724893 5751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:27.726267 5751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:27.738387 5756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:27.738356 5759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:27.738464 5757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:27.739176 5751 server_base.cc:1034] running on GCE node
I20260928 02:17:27.740001 5751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:27.741427 5751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:27.742688 5751 hybrid_clock.cc:648] HybridClock initialized: now 1790561847742630 us; error 59 us; skew 500 ppm
I20260928 02:17:27.743044 5751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:27 dist-test-slave-h37r krb5kdc[5738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561847, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:27.752868 5751 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:17:27.754505 5751 webserver.cc:460] Webserver started at http://127.4.140.254:34093/ using document root <none> and password file <none>
I20260928 02:17:27.755335 5751 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:27.755465 5751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:27.755811 5751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:27.758841 5751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "e5656ff10a184535a121fabbd68fb762"
format_stamp: "Formatted at 2026-09-28 02:17:27 on dist-test-slave-h37r"
server_key: "baf1ce991bf8839e9a7cb5828cb25f2b"
server_key_iv: "c7f7642044c368105494de66a0b52ea9"
server_key_version: "encryptionkey@0"
I20260928 02:17:27.759634 5751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "e5656ff10a184535a121fabbd68fb762"
format_stamp: "Formatted at 2026-09-28 02:17:27 on dist-test-slave-h37r"
server_key: "baf1ce991bf8839e9a7cb5828cb25f2b"
server_key_iv: "c7f7642044c368105494de66a0b52ea9"
server_key_version: "encryptionkey@0"
I20260928 02:17:27.765139 5751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 02:17:27.768796 5766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:27.770228 5751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:17:27.770440 5751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "e5656ff10a184535a121fabbd68fb762"
format_stamp: "Formatted at 2026-09-28 02:17:27 on dist-test-slave-h37r"
server_key: "baf1ce991bf8839e9a7cb5828cb25f2b"
server_key_iv: "c7f7642044c368105494de66a0b52ea9"
server_key_version: "encryptionkey@0"
I20260928 02:17:27.770694 5751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:27.792280 5751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:27.796598 5751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:27.796882 5751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:27.818871 5751 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:43303
I20260928 02:17:27.818883 5827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:43303 every 8 connection(s)
I20260928 02:17:27.820317 5751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:17:27.821828 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 5751
I20260928 02:17:27.822047 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:17:27.822947 4659 external_mini_cluster.cc:1428] Setting key 90dbe4b331d2a9b4b0569fa8a6987501
I20260928 02:17:27.825421 5828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 02:17:27 dist-test-slave-h37r krb5kdc[5738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:27.838665 5828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762: Bootstrap starting.
I20260928 02:17:27.841955 5828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:27.843055 5828 log.cc:824] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:27.845811 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:27.825091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56310 (local address 127.4.140.254:43303)
0928 02:17:27.825867 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:27.825880 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:27.825922 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:17:27.827526 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:27.827548 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:27.828036 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:27.828408 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:27.830093 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:27.831455 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:27.833150 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:27.833802 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:27.833935 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:27.838522 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:27.838531 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:27.838549 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:27.838588 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:27.841635 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:27.842402 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:27.842412 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:27.842420 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:27.842583 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:27.842974 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:27.842979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:27.842981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:27.843575 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:27.843902 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:27.844377 (+ 475us) server_negotiation.cc:299] Negotiation successful
0928 02:17:27.844785 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":381,"threads_started":1}
I20260928 02:17:27.846105 5828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762: No bootstrap required, opened a new log
I20260928 02:17:27.854084 5828 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5656ff10a184535a121fabbd68fb762" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43303 } }
I20260928 02:17:27.854616 5828 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:27.854733 5828 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5656ff10a184535a121fabbd68fb762, State: Initialized, Role: FOLLOWER
I20260928 02:17:27.855399 5828 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5656ff10a184535a121fabbd68fb762" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43303 } }
I20260928 02:17:27.855640 5828 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:17:27.855834 5828 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:17:27.856062 5828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:27.858184 5828 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5656ff10a184535a121fabbd68fb762" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43303 } }
I20260928 02:17:27.858695 5828 leader_election.cc:302] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5656ff10a184535a121fabbd68fb762; no voters:
I20260928 02:17:27.859313 5828 leader_election.cc:288] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:17:27.859457 5833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:27.860373 5833 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762 [term 1 LEADER]: Becoming Leader. State: Replica: e5656ff10a184535a121fabbd68fb762, State: Running, Role: LEADER
I20260928 02:17:27.861063 5833 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5656ff10a184535a121fabbd68fb762" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43303 } }
I20260928 02:17:27.861810 5828 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:17:27.864075 5835 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5656ff10a184535a121fabbd68fb762. Latest consensus state: current_term: 1 leader_uuid: "e5656ff10a184535a121fabbd68fb762" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5656ff10a184535a121fabbd68fb762" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43303 } } }
I20260928 02:17:27.864120 5834 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5656ff10a184535a121fabbd68fb762" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5656ff10a184535a121fabbd68fb762" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43303 } } }
I20260928 02:17:27.864534 5834 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762 [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:27.864543 5835 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762 [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:27.866590 5842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:17:27.872080 5842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:17:27.881968 5842 catalog_manager.cc:1347] Generated new cluster ID: c880dd9856454d45a26cd851fb30e439
I20260928 02:17:27.882123 5842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:17:27.896524 5842 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:17:27.898459 5842 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:17:27.910125 5842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762: Generated new TSK 0
I20260928 02:17:27.911305 5842 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
I20260928 02:17:27.968952 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43303
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:36101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:28.302068 5854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:28.302348 5854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:28.302513 5854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:28.309875 5854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:28.309993 5854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:28.310150 5854 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:17:28.310262 5854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:17:28.320338 5854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.140.254:43303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:28.321661 5854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:28.323067 5854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:28.336297 5859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:28.336345 5860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:28.336378 5862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:28.337977 5854 server_base.cc:1034] running on GCE node
I20260928 02:17:28.338681 5854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:28.339735 5854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:28.340994 5854 hybrid_clock.cc:648] HybridClock initialized: now 1790561848340926 us; error 45 us; skew 500 ppm
I20260928 02:17:28.341328 5854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:28 dist-test-slave-h37r krb5kdc[5738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:28.344847 5854 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:17:28.347257 5854 webserver.cc:460] Webserver started at http://127.4.140.193:46237/ using document root <none> and password file <none>
I20260928 02:17:28.348070 5854 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:28.348191 5854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:28.348536 5854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:28.351459 5854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "77bb4a4d5cb940ffa13f898b815371bf"
format_stamp: "Formatted at 2026-09-28 02:17:28 on dist-test-slave-h37r"
server_key: "21c7eb2672f1a166fe59843c5733c3d6"
server_key_iv: "bce284c0e5664c783ac54c2f1f11a98e"
server_key_version: "encryptionkey@0"
I20260928 02:17:28.352306 5854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "77bb4a4d5cb940ffa13f898b815371bf"
format_stamp: "Formatted at 2026-09-28 02:17:28 on dist-test-slave-h37r"
server_key: "21c7eb2672f1a166fe59843c5733c3d6"
server_key_iv: "bce284c0e5664c783ac54c2f1f11a98e"
server_key_version: "encryptionkey@0"
I20260928 02:17:28.357760 5854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:17:28.361538 5869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:28.363334 5854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:17:28.363528 5854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "77bb4a4d5cb940ffa13f898b815371bf"
format_stamp: "Formatted at 2026-09-28 02:17:28 on dist-test-slave-h37r"
server_key: "21c7eb2672f1a166fe59843c5733c3d6"
server_key_iv: "bce284c0e5664c783ac54c2f1f11a98e"
server_key_version: "encryptionkey@0"
I20260928 02:17:28.363777 5854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:28.401566 5854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:28.406052 5854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:28.406370 5854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:28.409408 5854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:28.409523 5854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:28.409670 5854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:28.409747 5854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:17:28 dist-test-slave-h37r krb5kdc[5738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:28.434095 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:28.410080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:46173 (local address 127.4.140.254:43303)
0928 02:17:28.413293 (+ 3213us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:28.413303 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:28.413332 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:17:28.414914 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:28.414920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:28.415031 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:28.415155 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:28.417203 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:28.418235 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:28.420951 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:28.421490 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:28.421659 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:28.426691 (+ 5032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:28.426696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:28.426699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:28.426732 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:28.430049 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:28.431086 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:28.431092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:28.431094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:28.431259 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:28.431815 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:28.431821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:28.431824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:28.432165 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:28.432359 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:28.433672 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0928 02:17:28.433843 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3057,"thread_start_us":2960,"threads_started":1}
I20260928 02:17:28.436611 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:28.411163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43303 (local address 127.4.140.193:46173)
0928 02:17:28.412264 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:28.412348 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:28.414622 (+ 2274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:28.415419 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:28.415428 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:28.416043 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:28.417008 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:28.417042 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:28.418442 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:28.418446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:28.420741 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:28.420760 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:28.421771 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:28.421787 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:28.422139 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:28.422943 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:28.422986 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:28.426399 (+ 3413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:28.430206 (+ 3807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:28.430216 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:28.430239 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:28.430936 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:28.431431 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:28.431438 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:28.431443 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:28.431675 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:28.432365 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:28.432380 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:28.433093 (+ 713us) client_negotiation.cc:769] Sending connection context
0928 02:17:28.433583 (+ 490us) client_negotiation.cc:240] Negotiation successful
0928 02:17:28.434425 (+ 842us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":402,"threads_started":1}
I20260928 02:17:28.461894 5947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:28.443221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43303 (local address 127.4.140.193:44867)
0928 02:17:28.444479 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:28.444508 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:28.444713 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:28.445130 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:28.445138 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:28.445877 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:28.446484 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:28.446502 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:28.447869 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:28.447879 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:28.450589 (+ 2710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:28.450618 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:28.451603 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:28.451614 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:28.451883 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:28.452894 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:28.452928 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:28.454407 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:28.458378 (+ 3971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:28.458383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:28.458387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:28.459034 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:28.459543 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:28.459557 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:28.459564 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:28.459793 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:28.460507 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:28.460512 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:28.460765 (+ 253us) client_negotiation.cc:769] Sending connection context
0928 02:17:28.460979 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 02:17:28.461380 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"thread_start_us":836,"threads_started":1}
I20260928 02:17:28.462421 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:28.443301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:44867 (local address 127.4.140.254:43303)
0928 02:17:28.443555 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:28.443564 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:28.444625 (+ 1061us) server_negotiation.cc:407] Connection header received
0928 02:17:28.444896 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:28.444902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:28.444995 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:28.445108 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:28.446711 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:28.447678 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:28.450842 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:28.451297 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:28.451466 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:28.454653 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:28.454658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:28.454662 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:28.454695 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:28.458196 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:28.459180 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:28.459186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:28.459190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:28.459391 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:28.460034 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:28.460042 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:28.460046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:28.460374 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:28.460538 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:28.461362 (+ 824us) server_negotiation.cc:299] Negotiation successful
0928 02:17:28.461625 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 02:17:28.466953 5854 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:41741
I20260928 02:17:28.466979 5989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:41741 every 8 connection(s)
I20260928 02:17:28.468645 5854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:17:28.469399 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 5854
I20260928 02:17:28.469645 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:17:28.470561 4659 external_mini_cluster.cc:1428] Setting key 0bedc10c58db8b4cd473ae167d19e9fc
I20260928 02:17:28.471094 5990 heartbeater.cc:347] Connected to a master server at 127.4.140.254:43303
I20260928 02:17:28.471347 5990 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:28.472077 5990 heartbeater.cc:511] Master 127.4.140.254:43303 requested a full tablet report, sending...
I20260928 02:17:28.474100 5783 ts_manager.cc:194] Registered new tserver with Master: 77bb4a4d5cb940ffa13f898b815371bf (127.4.140.193:41741)
I20260928 02:17:28.475682 5783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:46173
WARNING: no policy specified for kudu/127.4.140.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
I20260928 02:17:28.512732 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43303
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:36101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:28.836745 5994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:28.837019 5994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:28.837116 5994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:28.844682 5994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:28.844805 5994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:28.844897 5994 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:17:28.844980 5994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:17:28.855162 5994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.140.254:43303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:28.856554 5994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:28.857873 5994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:28.871697 5999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:28.871676 6002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:28.871723 6000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:28.872319 5994 server_base.cc:1034] running on GCE node
I20260928 02:17:28.873065 5994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:28.874135 5994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:28.875444 5994 hybrid_clock.cc:648] HybridClock initialized: now 1790561848875376 us; error 38 us; skew 500 ppm
I20260928 02:17:28.875857 5994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:28 dist-test-slave-h37r krb5kdc[5738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:28.879093 5994 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:17:28.880692 5994 webserver.cc:460] Webserver started at http://127.4.140.194:36099/ using document root <none> and password file <none>
I20260928 02:17:28.881428 5994 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:28.881556 5994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:28.881893 5994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:28.884905 5994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "bfc4b10f2ef44600bf1d82622687afaf"
format_stamp: "Formatted at 2026-09-28 02:17:28 on dist-test-slave-h37r"
server_key: "57c891a91b0b4c4405740ded6b848604"
server_key_iv: "ddaab85914768c7ec2d8adc25fca662c"
server_key_version: "encryptionkey@0"
I20260928 02:17:28.885711 5994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "bfc4b10f2ef44600bf1d82622687afaf"
format_stamp: "Formatted at 2026-09-28 02:17:28 on dist-test-slave-h37r"
server_key: "57c891a91b0b4c4405740ded6b848604"
server_key_iv: "ddaab85914768c7ec2d8adc25fca662c"
server_key_version: "encryptionkey@0"
I20260928 02:17:28.890815 5994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:17:28.894399 6009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:28.895722 5994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:17:28.895991 5994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "bfc4b10f2ef44600bf1d82622687afaf"
format_stamp: "Formatted at 2026-09-28 02:17:28 on dist-test-slave-h37r"
server_key: "57c891a91b0b4c4405740ded6b848604"
server_key_iv: "ddaab85914768c7ec2d8adc25fca662c"
server_key_version: "encryptionkey@0"
I20260928 02:17:28.896277 5994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:28.940461 5994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:28.944968 5994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:28.945271 5994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:28.948069 5994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:28.948177 5994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:28.948438 5994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:28.948510 5994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:17:28 dist-test-slave-h37r krb5kdc[5738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:28.977063 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:28.949046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:44983 (local address 127.4.140.254:43303)
0928 02:17:28.949246 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:28.949253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:28.950811 (+ 1558us) server_negotiation.cc:407] Connection header received
0928 02:17:28.953274 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:28.953284 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:28.953400 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:28.953527 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:28.956167 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:28.957237 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:28.960688 (+ 3451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:28.961464 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:28.961592 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:28.968395 (+ 6803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:28.968405 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:28.968412 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:28.968485 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:28.972508 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:28.973823 (+ 1315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:28.973837 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:28.973841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:28.974023 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:28.974716 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:28.974721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:28.974724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:28.975165 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:28.975404 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:28.976570 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0928 02:17:28.976789 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:17:28.979435 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:28.949751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43303 (local address 127.4.140.194:44983)
0928 02:17:28.950631 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:28.950678 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:28.952859 (+ 2181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:28.953851 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:28.953863 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:28.954782 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:28.955945 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:28.955984 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:28.957475 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:28.957487 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:28.960457 (+ 2970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:28.960480 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:28.962124 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:28.962138 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:28.962538 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:28.963592 (+ 1054us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:28.963640 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:28.968050 (+ 4410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:28.972763 (+ 4713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:28.972778 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:28.972809 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:28.973653 (+ 844us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:28.974153 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:28.974159 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:28.974163 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:28.974573 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:28.975354 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:28.975371 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:28.976020 (+ 649us) client_negotiation.cc:769] Sending connection context
0928 02:17:28.976423 (+ 403us) client_negotiation.cc:240] Negotiation successful
0928 02:17:28.977193 (+ 770us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":366,"threads_started":1}
I20260928 02:17:29.001438 5994 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:43935
I20260928 02:17:29.001397 6128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:43935 every 8 connection(s)
I20260928 02:17:29.002879 5994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:17:29.003454 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:28.986588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:54955 (local address 127.4.140.254:43303)
0928 02:17:28.986880 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:28.986889 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:28.986990 (+ 101us) server_negotiation.cc:407] Connection header received
0928 02:17:28.987165 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:28.987174 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:28.987290 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:28.987425 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:28.988837 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:28.990289 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:28.992874 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:28.993328 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:28.993481 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:28.996535 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:28.996544 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:28.996552 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:28.996600 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:29.000007 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:29.001400 (+ 1393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:29.001405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:29.001408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:29.001582 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:29.001962 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:29.001967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:29.001970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:29.002299 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:29.002453 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:29.002875 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 02:17:29.003093 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 02:17:29.003558 6107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:28.986243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43303 (local address 127.4.140.194:54955)
0928 02:17:28.986865 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:28.986888 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:28.987037 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:28.987461 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:28.987465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:28.987999 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:28.988602 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:28.988648 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:28.990473 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:28.990478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:28.992687 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:28.992700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:28.993550 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:28.993559 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:28.993778 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:28.994650 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:28.994687 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:28.996296 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:29.000210 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:29.000215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:29.000219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:29.001241 (+ 1022us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:29.001690 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:29.001696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:29.001700 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:29.001842 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:29.002418 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:29.002423 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:29.002619 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 02:17:29.002768 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 02:17:29.003118 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":418,"threads_started":1}
I20260928 02:17:29.005292 6129 heartbeater.cc:347] Connected to a master server at 127.4.140.254:43303
I20260928 02:17:29.005946 6129 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:29.006812 6129 heartbeater.cc:511] Master 127.4.140.254:43303 requested a full tablet report, sending...
I20260928 02:17:29.008708 5783 ts_manager.cc:194] Registered new tserver with Master: bfc4b10f2ef44600bf1d82622687afaf (127.4.140.194:43935)
I20260928 02:17:29.009558 5783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:44983
I20260928 02:17:29.011422 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 5994
I20260928 02:17:29.011607 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:17:29.012138 4659 external_mini_cluster.cc:1428] Setting key 7de2bb833121666e2f5e27c741aeac2e
WARNING: no policy specified for kudu/127.4.140.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
I20260928 02:17:29.052402 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43303
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:36101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:29.377624 6133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:29.377926 6133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:29.378072 6133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:29.385867 6133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:29.386019 6133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:29.386184 6133 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:17:29.386327 6133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:17:29.396550 6133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.140.254:43303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:29.397872 6133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:29.399211 6133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:29.412779 6139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:29.412782 6138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:29.412782 6141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:29.413800 6133 server_base.cc:1034] running on GCE node
I20260928 02:17:29.414516 6133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:29.415470 6133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:29.416693 6133 hybrid_clock.cc:648] HybridClock initialized: now 1790561849416631 us; error 39 us; skew 500 ppm
I20260928 02:17:29.417011 6133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:29 dist-test-slave-h37r krb5kdc[5738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561849, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:29.420131 6133 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:17:29.421903 6133 webserver.cc:460] Webserver started at http://127.4.140.195:40365/ using document root <none> and password file <none>
I20260928 02:17:29.422765 6133 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:29.422886 6133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:29.423203 6133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:29.426157 6133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "414aa59e2e8f40e78885ff00831c4614"
format_stamp: "Formatted at 2026-09-28 02:17:29 on dist-test-slave-h37r"
server_key: "f42cd728c7bc71464d20c58e189d3e0f"
server_key_iv: "4dd5684eb99efafb877932594d967969"
server_key_version: "encryptionkey@0"
I20260928 02:17:29.426963 6133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "414aa59e2e8f40e78885ff00831c4614"
format_stamp: "Formatted at 2026-09-28 02:17:29 on dist-test-slave-h37r"
server_key: "f42cd728c7bc71464d20c58e189d3e0f"
server_key_iv: "4dd5684eb99efafb877932594d967969"
server_key_version: "encryptionkey@0"
I20260928 02:17:29.432189 6133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:17:29.436579 6148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:29.438025 6133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:17:29.438206 6133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "414aa59e2e8f40e78885ff00831c4614"
format_stamp: "Formatted at 2026-09-28 02:17:29 on dist-test-slave-h37r"
server_key: "f42cd728c7bc71464d20c58e189d3e0f"
server_key_iv: "4dd5684eb99efafb877932594d967969"
server_key_version: "encryptionkey@0"
I20260928 02:17:29.438452 6133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:29.464144 6133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:29.469257 6133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:29.469576 6133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:29.472927 6133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:29.473042 6133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:29.473166 6133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:29.473241 6133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:29.478636 5990 heartbeater.cc:503] Master 127.4.140.254:43303 was elected leader, sending a full tablet report...
Sep 28 02:17:29 dist-test-slave-h37r krb5kdc[5738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561849, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:29.497956 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.473892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:40321 (local address 127.4.140.254:43303)
0928 02:17:29.474084 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:29.474090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:29.475732 (+ 1642us) server_negotiation.cc:407] Connection header received
0928 02:17:29.478254 (+ 2522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:29.478260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:29.478345 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:29.478462 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:29.480395 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.481364 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.483804 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.484381 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.484548 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.489955 (+ 5407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:29.489960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:29.489963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:29.489995 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:29.493495 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:29.494604 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:29.494611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:29.494615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:29.494785 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:29.495454 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:29.495458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:29.495460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:29.495850 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:29.496018 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:29.497221 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0928 02:17:29.497564 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:17:29.499488 6157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.474591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43303 (local address 127.4.140.195:40321)
0928 02:17:29.475572 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:29.475609 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:29.477868 (+ 2259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:29.478695 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:29.478702 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:29.479479 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:29.480232 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:29.480250 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.481499 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.481510 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:29.483650 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:29.483663 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.484659 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.484669 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:29.485065 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.485922 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:29.485961 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:29.489625 (+ 3664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:29.493651 (+ 4026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:29.493663 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:29.493682 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:29.494379 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:29.494916 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:29.494930 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:29.494932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:29.495266 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:29.496082 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:29.496112 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:29.496659 (+ 547us) client_negotiation.cc:769] Sending connection context
0928 02:17:29.497059 (+ 400us) client_negotiation.cc:240] Negotiation successful
0928 02:17:29.497944 (+ 885us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":465,"threads_started":1}
I20260928 02:17:29.520808 6133 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:34197
I20260928 02:17:29.520833 6267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:34197 every 8 connection(s)
I20260928 02:17:29.522085 6133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:17:29.523521 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.507111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:43947 (local address 127.4.140.254:43303)
0928 02:17:29.507354 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:29.507361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:29.508596 (+ 1235us) server_negotiation.cc:407] Connection header received
0928 02:17:29.508793 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:29.508801 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:29.508906 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:29.509026 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:29.510592 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.511792 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.513802 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.514565 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.514714 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.517431 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:29.517436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:29.517440 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:29.517470 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:29.520732 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:29.521484 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:29.521489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:29.521494 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:29.521620 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:29.522000 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:29.522010 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:29.522012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:29.522322 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:29.522469 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:29.522913 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 02:17:29.523132 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:17:29.523963 6233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.507418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43303 (local address 127.4.140.195:43947)
0928 02:17:29.508451 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:29.508476 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:29.508636 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:29.509087 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:29.509094 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:29.509793 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:29.510339 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:29.510351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.512008 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.512013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:29.513588 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:29.513600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.514762 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.514771 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:29.515045 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.515793 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:29.515837 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:29.517193 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:29.520877 (+ 3684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:29.520882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:29.520886 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:29.521355 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:29.521730 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:29.521736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:29.521739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:29.521882 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:29.522446 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:29.522451 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:29.522645 (+ 194us) client_negotiation.cc:769] Sending connection context
0928 02:17:29.522804 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 02:17:29.523167 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"thread_start_us":685,"threads_started":1}
I20260928 02:17:29.524525 6268 heartbeater.cc:347] Connected to a master server at 127.4.140.254:43303
I20260928 02:17:29.524986 6268 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:29.525611 6268 heartbeater.cc:511] Master 127.4.140.254:43303 requested a full tablet report, sending...
I20260928 02:17:29.527002 5783 ts_manager.cc:194] Registered new tserver with Master: 414aa59e2e8f40e78885ff00831c4614 (127.4.140.195:34197)
I20260928 02:17:29.528112 5783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:40321
I20260928 02:17:29.530728 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 6133
I20260928 02:17:29.530941 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:17:29.531411 4659 external_mini_cluster.cc:1428] Setting key de06fd02ed965b6c670aefa432b71425
I20260928 02:17:29.534699 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:17:29 dist-test-slave-h37r krb5kdc[5738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561849, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:17:29 dist-test-slave-h37r krb5kdc[5738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561849, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:29.581228 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.567080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.4.140.254:43303)
0928 02:17:29.567253 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:29.567257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:29.567613 (+ 356us) server_negotiation.cc:407] Connection header received
0928 02:17:29.567816 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:29.567819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:29.567885 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:29.568043 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:29.569417 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.570124 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.571364 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.571783 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.571878 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.575796 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:29.575801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:29.575805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:29.575838 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:29.578777 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:29.579374 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:29.579379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:29.579381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:29.579503 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:29.579867 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:29.579870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:29.579872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:29.580211 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:29.580410 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:29.580726 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 02:17:29.580912 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:17:29.585637 5783 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:17:29.592407 5783 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:17:29.618477 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.605428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51038 (local address 127.4.140.195:34197)
0928 02:17:29.610914 (+ 5486us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:29.610923 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:29.610952 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:17:29.611027 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:29.611032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:29.611227 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:29.611400 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:29.612247 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.613715 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.615971 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.617292 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.617488 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.617735 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:29.617960 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 02:17:29.618121 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5323,"thread_start_us":5139,"threads_started":1}
I20260928 02:17:29.618530 6285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.604777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:34197 (local address 127.0.0.1:51038)
0928 02:17:29.607058 (+ 2281us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:29.607076 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:29.607193 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:29.611394 (+ 4201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:29.611399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:29.611418 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:29.612036 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:29.612046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.613856 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.613863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:29.615792 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:29.615804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.617517 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.617523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:29.617717 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.617769 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 02:17:29.617931 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 02:17:29.618120 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2144,"mutex_wait_us":86,"thread_start_us":360,"threads_started":1}
I20260928 02:17:29.619717 6286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.607044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50262 (local address 127.4.140.194:43935)
0928 02:17:29.610130 (+ 3086us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:29.610143 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:29.610653 (+ 510us) server_negotiation.cc:407] Connection header received
0928 02:17:29.610928 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:29.610938 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:29.611163 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:29.611374 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:29.612418 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.613807 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.616751 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.618119 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.618260 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.618446 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:29.619306 (+ 860us) server_negotiation.cc:299] Negotiation successful
0928 02:17:29.619460 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2943,"thread_start_us":2825,"threads_started":1}
I20260928 02:17:29.619817 6282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.606498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:43935 (local address 127.0.0.1:50262)
0928 02:17:29.610550 (+ 4052us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:29.610567 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:29.610744 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:29.611612 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:29.611617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:29.611646 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:29.612225 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:29.612238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.613924 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.613928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:29.616574 (+ 2646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:29.616583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.618624 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.618631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:29.619042 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.619125 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 02:17:29.619220 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 02:17:29.619404 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3948,"mutex_wait_us":288,"thread_start_us":715,"threads_started":1}
I20260928 02:17:29.619877 6281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.604811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.4.140.193:41741)
0928 02:17:29.610505 (+ 5694us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:29.610515 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:29.610545 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:17:29.610671 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:29.610681 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:29.610957 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:29.611157 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:29.611955 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.613543 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.616765 (+ 3222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.618119 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.618255 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.618446 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:29.619364 (+ 918us) server_negotiation.cc:299] Negotiation successful
0928 02:17:29.619563 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5522,"thread_start_us":5400,"threads_started":1}
I20260928 02:17:29.620671 6283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.604713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:41741 (local address 127.0.0.1:53388)
0928 02:17:29.605505 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:29.605540 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:29.605692 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:29.611158 (+ 5466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:29.611165 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:29.611193 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:29.611628 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:29.611641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.613687 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.613693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:29.616549 (+ 2856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:29.616570 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.618638 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.618643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:29.619050 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.619131 (+ 81us) client_negotiation.cc:769] Sending connection context
0928 02:17:29.619223 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 02:17:29.620214 (+ 991us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":1130,"thread_start_us":5571,"threads_started":1}
I20260928 02:17:29.622809 6198 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab8f4462d424d2c870045c623c1fb0a (DEFAULT_TABLE table=test-table [id=c1ca36c7b8e54be4a0597a9041f51a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:29.623502 6059 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab8f4462d424d2c870045c623c1fb0a (DEFAULT_TABLE table=test-table [id=c1ca36c7b8e54be4a0597a9041f51a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:29.623768 5920 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab8f4462d424d2c870045c623c1fb0a (DEFAULT_TABLE table=test-table [id=c1ca36c7b8e54be4a0597a9041f51a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:29.624342 6198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ab8f4462d424d2c870045c623c1fb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:29.624796 6059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ab8f4462d424d2c870045c623c1fb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:29.625053 5920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ab8f4462d424d2c870045c623c1fb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:29.636875 6287 tablet_bootstrap.cc:492] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf: Bootstrap starting.
I20260928 02:17:29.640528 6287 tablet_bootstrap.cc:654] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:29.641597 6287 log.cc:824] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:29.643054 6288 tablet_bootstrap.cc:492] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf: Bootstrap starting.
I20260928 02:17:29.645066 6289 tablet_bootstrap.cc:492] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614: Bootstrap starting.
I20260928 02:17:29.645372 6287 tablet_bootstrap.cc:492] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf: No bootstrap required, opened a new log
I20260928 02:17:29.645661 6287 ts_tablet_manager.cc:1397] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 02:17:29.649006 6289 tablet_bootstrap.cc:654] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:29.650309 6288 tablet_bootstrap.cc:654] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:29.650338 6289 log.cc:824] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:29.652200 6288 log.cc:824] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:29.653725 6289 tablet_bootstrap.cc:492] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614: No bootstrap required, opened a new log
I20260928 02:17:29.654016 6289 ts_tablet_manager.cc:1397] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 02:17:29.654757 6288 tablet_bootstrap.cc:492] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf: No bootstrap required, opened a new log
I20260928 02:17:29.655012 6288 ts_tablet_manager.cc:1397] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260928 02:17:29.657577 6287 raft_consensus.cc:348] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } }
I20260928 02:17:29.658056 6287 raft_consensus.cc:374] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:29.658159 6287 raft_consensus.cc:729] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bb4a4d5cb940ffa13f898b815371bf, State: Initialized, Role: FOLLOWER
I20260928 02:17:29.658877 6287 consensus_queue.cc:260] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } }
I20260928 02:17:29.660785 6287 ts_tablet_manager.cc:1428] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260928 02:17:29.663100 6288 raft_consensus.cc:348] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } }
I20260928 02:17:29.663527 6288 raft_consensus.cc:374] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:29.663637 6288 raft_consensus.cc:729] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfc4b10f2ef44600bf1d82622687afaf, State: Initialized, Role: FOLLOWER
I20260928 02:17:29.664294 6288 consensus_queue.cc:260] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } }
I20260928 02:17:29.665479 6129 heartbeater.cc:503] Master 127.4.140.254:43303 was elected leader, sending a full tablet report...
I20260928 02:17:29.665369 6289 raft_consensus.cc:348] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } }
I20260928 02:17:29.665835 6289 raft_consensus.cc:374] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:29.665932 6289 raft_consensus.cc:729] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 414aa59e2e8f40e78885ff00831c4614, State: Initialized, Role: FOLLOWER
I20260928 02:17:29.666015 6288 ts_tablet_manager.cc:1428] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260928 02:17:29.666625 6289 consensus_queue.cc:260] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } }
I20260928 02:17:29.668285 6268 heartbeater.cc:503] Master 127.4.140.254:43303 was elected leader, sending a full tablet report...
I20260928 02:17:29.669210 6289 ts_tablet_manager.cc:1428] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260928 02:17:29.699582 6293 raft_consensus.cc:482] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:17:29.699868 6293 raft_consensus.cc:504] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } }
I20260928 02:17:29.701769 6293 leader_election.cc:288] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 414aa59e2e8f40e78885ff00831c4614 (127.4.140.195:34197), bfc4b10f2ef44600bf1d82622687afaf (127.4.140.194:43935)
I20260928 02:17:29.709268 6286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.702274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:33931 (local address 127.4.140.194:43935)
0928 02:17:29.702487 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:29.702496 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:29.703358 (+ 862us) server_negotiation.cc:407] Connection header received
0928 02:17:29.703546 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:29.703553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:29.703615 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:29.703731 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:29.704442 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.705572 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.707519 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.708340 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.708534 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.708609 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:29.708888 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 02:17:29.709007 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 02:17:29.709304 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.702283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:53623 (local address 127.4.140.195:34197)
0928 02:17:29.702490 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:29.702499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:29.703265 (+ 766us) server_negotiation.cc:407] Connection header received
0928 02:17:29.703504 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:29.703513 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:29.703597 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:29.703660 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:29.704552 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.705572 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.707525 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.708388 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.708572 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.708653 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:29.708933 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 02:17:29.709052 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:17:29.709444 6297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.702262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:43935 (local address 127.4.140.193:33931)
0928 02:17:29.703261 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:29.703280 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:29.703421 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:29.703740 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:29.703744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:29.703777 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:29.704199 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:29.704208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.705763 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.705767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:29.707360 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:29.707372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.708562 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.708569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:29.708730 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.708754 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:17:29.708861 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:17:29.708989 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":170,"thread_start_us":470,"threads_started":1}
I20260928 02:17:29.709467 6296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.702262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:34197 (local address 127.4.140.193:53623)
0928 02:17:29.703138 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:29.703161 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:29.703318 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:29.703881 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:29.703885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:29.703944 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:29.704401 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:29.704414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.705715 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.705719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:29.707357 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:29.707372 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.708609 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.708614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:29.708769 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.708790 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:17:29.708898 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 02:17:29.709017 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":84,"thread_start_us":462,"threads_started":1}
I20260928 02:17:29.710538 6079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab8f4462d424d2c870045c623c1fb0a" candidate_uuid: "77bb4a4d5cb940ffa13f898b815371bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfc4b10f2ef44600bf1d82622687afaf" is_pre_election: true
I20260928 02:17:29.710713 6218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab8f4462d424d2c870045c623c1fb0a" candidate_uuid: "77bb4a4d5cb940ffa13f898b815371bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "414aa59e2e8f40e78885ff00831c4614" is_pre_election: true
I20260928 02:17:29.711057 6079 raft_consensus.cc:2454] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bb4a4d5cb940ffa13f898b815371bf in term 0.
I20260928 02:17:29.711279 6218 raft_consensus.cc:2454] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bb4a4d5cb940ffa13f898b815371bf in term 0.
I20260928 02:17:29.711807 5870 leader_election.cc:302] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bb4a4d5cb940ffa13f898b815371bf, bfc4b10f2ef44600bf1d82622687afaf; no voters:
I20260928 02:17:29.712235 6293 raft_consensus.cc:2781] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:17:29.712383 6293 raft_consensus.cc:482] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:17:29.712491 6293 raft_consensus.cc:3037] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:29.714565 6293 raft_consensus.cc:504] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } }
I20260928 02:17:29.715167 6293 leader_election.cc:288] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [CANDIDATE]: Term 1 election: Requested vote from peers 414aa59e2e8f40e78885ff00831c4614 (127.4.140.195:34197), bfc4b10f2ef44600bf1d82622687afaf (127.4.140.194:43935)
I20260928 02:17:29.715678 6218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab8f4462d424d2c870045c623c1fb0a" candidate_uuid: "77bb4a4d5cb940ffa13f898b815371bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "414aa59e2e8f40e78885ff00831c4614"
I20260928 02:17:29.715715 6079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab8f4462d424d2c870045c623c1fb0a" candidate_uuid: "77bb4a4d5cb940ffa13f898b815371bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfc4b10f2ef44600bf1d82622687afaf"
I20260928 02:17:29.715876 6218 raft_consensus.cc:3037] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:29.715986 6079 raft_consensus.cc:3037] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:29.718170 6218 raft_consensus.cc:2454] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bb4a4d5cb940ffa13f898b815371bf in term 1.
I20260928 02:17:29.718308 6079 raft_consensus.cc:2454] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bb4a4d5cb940ffa13f898b815371bf in term 1.
I20260928 02:17:29.718619 5871 leader_election.cc:302] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 414aa59e2e8f40e78885ff00831c4614, 77bb4a4d5cb940ffa13f898b815371bf; no voters:
I20260928 02:17:29.718988 6293 raft_consensus.cc:2781] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:29.719461 6293 raft_consensus.cc:686] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [term 1 LEADER]: Becoming Leader. State: Replica: 77bb4a4d5cb940ffa13f898b815371bf, State: Running, Role: LEADER
I20260928 02:17:29.720069 6293 consensus_queue.cc:237] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } }
W20260928 02:17:29.721390 5991 tablet.cc:2369] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:17:29.724756 5782 catalog_manager.cc:5495] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 77bb4a4d5cb940ffa13f898b815371bf (127.4.140.193). New cstate: current_term: 1 leader_uuid: "77bb4a4d5cb940ffa13f898b815371bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } health_report { overall_health: UNKNOWN } } }
I20260928 02:17:29.748356 6281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.742476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53396 (local address 127.4.140.193:41741)
0928 02:17:29.742667 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:29.742671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:29.742694 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:17:29.743119 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:29.743123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:29.743215 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:29.743427 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:17:29.744007 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.744760 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.746089 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.746496 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.746599 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.747106 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:17:29.747705 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:17:29.747830 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:29.748034 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 02:17:29.748118 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 02:17:29.756281 6130 tablet.cc:2369] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:17:29.757104 6079 raft_consensus.cc:1261] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf [term 1 FOLLOWER]: Refusing update from remote peer 77bb4a4d5cb940ffa13f898b815371bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:17:29.757097 6218 raft_consensus.cc:1261] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614 [term 1 FOLLOWER]: Refusing update from remote peer 77bb4a4d5cb940ffa13f898b815371bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:17:29.758327 6298 consensus_queue.cc:1035] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:17:29.758713 6293 consensus_queue.cc:1035] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:17:29.776484 6303 mvcc.cc:205] Tried to move back new op lower bound from 7334141336593547264 to 7334141336457134080. Current Snapshot: MvccSnapshot[applied={T|T < 7334141336593547264}]
I20260928 02:17:29.792620 5782 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:17:29.792951 5782 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:17:29.796567 5782 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762: Sending DeleteTablet for 3 replicas of tablet 2ab8f4462d424d2c870045c623c1fb0a
I20260928 02:17:29.797897 6059 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ab8f4462d424d2c870045c623c1fb0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:29 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:50262
I20260928 02:17:29.797839 5920 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ab8f4462d424d2c870045c623c1fb0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:29 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:53388
I20260928 02:17:29.797926 6198 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ab8f4462d424d2c870045c623c1fb0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:29 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:51038
I20260928 02:17:29.804706 5782 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:17:29.805640 5782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:17:29.811403 6313 tablet_replica.cc:330] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614: stopping tablet replica
I20260928 02:17:29.812124 6313 raft_consensus.cc:2229] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:17:29.812788 6313 raft_consensus.cc:2258] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:29.817350 5919 tablet_service.cc:1467] Processing CreateTablet for tablet 7b5c40cdc3cf42528c72575bfdc69e7c (DEFAULT_TABLE table=test-table [id=681384452b7f4bb0a1ebe0a9de993395]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:29.818284 5919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b5c40cdc3cf42528c72575bfdc69e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:29.820986 6058 tablet_service.cc:1467] Processing CreateTablet for tablet 7b5c40cdc3cf42528c72575bfdc69e7c (DEFAULT_TABLE table=test-table [id=681384452b7f4bb0a1ebe0a9de993395]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:29.821588 6058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b5c40cdc3cf42528c72575bfdc69e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:29.825759 6313 ts_tablet_manager.cc:1905] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:29.827194 6287 tablet_bootstrap.cc:492] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf: Bootstrap starting.
I20260928 02:17:29.827862 6198 tablet_service.cc:1467] Processing CreateTablet for tablet 7b5c40cdc3cf42528c72575bfdc69e7c (DEFAULT_TABLE table=test-table [id=681384452b7f4bb0a1ebe0a9de993395]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:29.828366 6198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b5c40cdc3cf42528c72575bfdc69e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:29.831048 6287 tablet_bootstrap.cc:654] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:29.833590 6313 ts_tablet_manager.cc:1918] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:17:29.833798 6313 log.cc:1196] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/2ab8f4462d424d2c870045c623c1fb0a
I20260928 02:17:29.834384 6288 tablet_bootstrap.cc:492] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf: Bootstrap starting.
I20260928 02:17:29.836715 6288 tablet_bootstrap.cc:654] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:29.840907 6287 tablet_bootstrap.cc:492] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf: No bootstrap required, opened a new log
I20260928 02:17:29.841214 6287 ts_tablet_manager.cc:1397] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20260928 02:17:29.842201 6287 raft_consensus.cc:348] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } }
I20260928 02:17:29.842667 6287 raft_consensus.cc:374] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:29.842839 6287 raft_consensus.cc:729] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bb4a4d5cb940ffa13f898b815371bf, State: Initialized, Role: FOLLOWER
I20260928 02:17:29.843245 6287 consensus_queue.cc:260] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } }
I20260928 02:17:29.844240 6289 tablet_bootstrap.cc:492] T 7b5c40cdc3cf42528c72575bfdc69e7c P 414aa59e2e8f40e78885ff00831c4614: Bootstrap starting.
I20260928 02:17:29.845170 6313 ts_tablet_manager.cc:1939] T 2ab8f4462d424d2c870045c623c1fb0a P 414aa59e2e8f40e78885ff00831c4614: Deleting consensus metadata
I20260928 02:17:29.845371 6287 ts_tablet_manager.cc:1428] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260928 02:17:29.846711 6289 tablet_bootstrap.cc:654] T 7b5c40cdc3cf42528c72575bfdc69e7c P 414aa59e2e8f40e78885ff00831c4614: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:29.847241 5768 catalog_manager.cc:4841] TS 414aa59e2e8f40e78885ff00831c4614 (127.4.140.195:34197): tablet 2ab8f4462d424d2c870045c623c1fb0a (table test-table [id=c1ca36c7b8e54be4a0597a9041f51a68]) successfully deleted
I20260928 02:17:29.851332 6311 tablet_replica.cc:330] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf: stopping tablet replica
I20260928 02:17:29.851926 6311 raft_consensus.cc:2229] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:17:29.852375 6298 raft_consensus.cc:482] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:17:29.852594 6298 raft_consensus.cc:504] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } }
I20260928 02:17:29.853371 6298 leader_election.cc:288] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfc4b10f2ef44600bf1d82622687afaf (127.4.140.194:43935), 414aa59e2e8f40e78885ff00831c4614 (127.4.140.195:34197)
I20260928 02:17:29.854008 6079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5c40cdc3cf42528c72575bfdc69e7c" candidate_uuid: "77bb4a4d5cb940ffa13f898b815371bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfc4b10f2ef44600bf1d82622687afaf" is_pre_election: true
I20260928 02:17:29.854015 6218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5c40cdc3cf42528c72575bfdc69e7c" candidate_uuid: "77bb4a4d5cb940ffa13f898b815371bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "414aa59e2e8f40e78885ff00831c4614" is_pre_election: true
W20260928 02:17:29.854859 5870 leader_election.cc:341] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bfc4b10f2ef44600bf1d82622687afaf (127.4.140.194:43935): Illegal state: must be running to vote when last-logged opid is not known
W20260928 02:17:29.855317 5871 leader_election.cc:341] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 414aa59e2e8f40e78885ff00831c4614 (127.4.140.195:34197): Illegal state: must be running to vote when last-logged opid is not known
I20260928 02:17:29.855502 5871 leader_election.cc:302] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77bb4a4d5cb940ffa13f898b815371bf; no voters: 414aa59e2e8f40e78885ff00831c4614, bfc4b10f2ef44600bf1d82622687afaf
I20260928 02:17:29.855545 6311 raft_consensus.cc:2258] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:29.856174 6298 raft_consensus.cc:2726] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:17:29.861253 6312 tablet_replica.cc:330] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf: stopping tablet replica
I20260928 02:17:29.861351 6288 tablet_bootstrap.cc:492] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf: No bootstrap required, opened a new log
I20260928 02:17:29.861521 6288 ts_tablet_manager.cc:1397] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.000s
I20260928 02:17:29.861927 6289 tablet_bootstrap.cc:492] T 7b5c40cdc3cf42528c72575bfdc69e7c P 414aa59e2e8f40e78885ff00831c4614: No bootstrap required, opened a new log
I20260928 02:17:29.862053 6289 ts_tablet_manager.cc:1397] T 7b5c40cdc3cf42528c72575bfdc69e7c P 414aa59e2e8f40e78885ff00831c4614: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.009s
I20260928 02:17:29.862416 6288 raft_consensus.cc:348] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } }
I20260928 02:17:29.862653 6288 raft_consensus.cc:374] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:29.862762 6288 raft_consensus.cc:729] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfc4b10f2ef44600bf1d82622687afaf, State: Initialized, Role: FOLLOWER
I20260928 02:17:29.862875 6289 raft_consensus.cc:348] T 7b5c40cdc3cf42528c72575bfdc69e7c P 414aa59e2e8f40e78885ff00831c4614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } }
I20260928 02:17:29.862987 6311 ts_tablet_manager.cc:1905] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:29.863088 6289 raft_consensus.cc:374] T 7b5c40cdc3cf42528c72575bfdc69e7c P 414aa59e2e8f40e78885ff00831c4614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:29.863234 6289 raft_consensus.cc:729] T 7b5c40cdc3cf42528c72575bfdc69e7c P 414aa59e2e8f40e78885ff00831c4614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 414aa59e2e8f40e78885ff00831c4614, State: Initialized, Role: FOLLOWER
I20260928 02:17:29.862995 6288 consensus_queue.cc:260] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } }
I20260928 02:17:29.863844 6288 ts_tablet_manager.cc:1428] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:17:29.863704 6289 consensus_queue.cc:260] T 7b5c40cdc3cf42528c72575bfdc69e7c P 414aa59e2e8f40e78885ff00831c4614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } }
I20260928 02:17:29.865420 6312 raft_consensus.cc:2229] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:17:29.866223 6312 raft_consensus.cc:2258] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:29.869987 6312 ts_tablet_manager.cc:1905] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:29.870592 6289 ts_tablet_manager.cc:1428] T 7b5c40cdc3cf42528c72575bfdc69e7c P 414aa59e2e8f40e78885ff00831c4614: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260928 02:17:29.875281 6311 ts_tablet_manager.cc:1918] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:17:29.875514 6311 log.cc:1196] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/2ab8f4462d424d2c870045c623c1fb0a
I20260928 02:17:29.876492 6311 ts_tablet_manager.cc:1939] T 2ab8f4462d424d2c870045c623c1fb0a P bfc4b10f2ef44600bf1d82622687afaf: Deleting consensus metadata
I20260928 02:17:29.877733 6312 ts_tablet_manager.cc:1918] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:17:29.877944 6312 log.cc:1196] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/2ab8f4462d424d2c870045c623c1fb0a
I20260928 02:17:29.878527 6312 ts_tablet_manager.cc:1939] T 2ab8f4462d424d2c870045c623c1fb0a P 77bb4a4d5cb940ffa13f898b815371bf: Deleting consensus metadata
I20260928 02:17:29.878537 5767 catalog_manager.cc:4841] TS bfc4b10f2ef44600bf1d82622687afaf (127.4.140.194:43935): tablet 2ab8f4462d424d2c870045c623c1fb0a (table test-table [id=c1ca36c7b8e54be4a0597a9041f51a68]) successfully deleted
I20260928 02:17:29.880072 5769 catalog_manager.cc:4841] TS 77bb4a4d5cb940ffa13f898b815371bf (127.4.140.193:41741): tablet 2ab8f4462d424d2c870045c623c1fb0a (table test-table [id=c1ca36c7b8e54be4a0597a9041f51a68]) successfully deleted
I20260928 02:17:29.889694 6294 raft_consensus.cc:482] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:17:29.889937 6294 raft_consensus.cc:504] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } }
I20260928 02:17:29.892095 6294 leader_election.cc:288] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77bb4a4d5cb940ffa13f898b815371bf (127.4.140.193:41741), 414aa59e2e8f40e78885ff00831c4614 (127.4.140.195:34197)
I20260928 02:17:29.901546 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.892586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:38407 (local address 127.4.140.195:34197)
0928 02:17:29.892782 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:29.892790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:29.893802 (+ 1012us) server_negotiation.cc:407] Connection header received
0928 02:17:29.893999 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:29.894005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:29.894094 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:29.894150 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:29.895089 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.896417 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.899124 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.900214 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.900412 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.900508 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:29.901072 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 02:17:29.901229 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:17:29.901629 6317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.892534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:34197 (local address 127.4.140.194:38407)
0928 02:17:29.893673 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:29.893692 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:29.893820 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:29.894327 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:29.894330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:29.894356 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:29.894836 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:29.894848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.896589 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.896595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:29.898919 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:29.898928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.900495 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.900501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:29.900864 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.900903 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 02:17:29.901018 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 02:17:29.901186 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":28,"thread_start_us":377,"threads_started":1}
I20260928 02:17:29.901759 6281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.892732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:38879 (local address 127.4.140.193:41741)
0928 02:17:29.892987 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:29.892995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:29.893377 (+ 382us) server_negotiation.cc:407] Connection header received
0928 02:17:29.893695 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:29.893702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:29.893805 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:29.893900 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:29.894595 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.896001 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.899097 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.900596 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.900779 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.900868 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:29.901236 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 02:17:29.901410 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260928 02:17:29.902623 6318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.892294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:41741 (local address 127.4.140.194:38879)
0928 02:17:29.893260 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:29.893282 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:29.893508 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:29.893949 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:29.893955 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:29.893985 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:29.894401 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:29.894415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.896189 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.896195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:29.898888 (+ 2693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:29.898907 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.900848 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.900854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:29.901052 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.901084 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:17:29.902035 (+ 951us) client_negotiation.cc:240] Negotiation successful
0928 02:17:29.902205 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":83,"thread_start_us":1064,"threads_started":1}
I20260928 02:17:29.902678 6218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5c40cdc3cf42528c72575bfdc69e7c" candidate_uuid: "bfc4b10f2ef44600bf1d82622687afaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "414aa59e2e8f40e78885ff00831c4614" is_pre_election: true
I20260928 02:17:29.902976 6218 raft_consensus.cc:2454] T 7b5c40cdc3cf42528c72575bfdc69e7c P 414aa59e2e8f40e78885ff00831c4614 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfc4b10f2ef44600bf1d82622687afaf in term 0.
I20260928 02:17:29.903539 6011 leader_election.cc:302] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 414aa59e2e8f40e78885ff00831c4614, bfc4b10f2ef44600bf1d82622687afaf; no voters:
I20260928 02:17:29.903658 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5c40cdc3cf42528c72575bfdc69e7c" candidate_uuid: "bfc4b10f2ef44600bf1d82622687afaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bb4a4d5cb940ffa13f898b815371bf" is_pre_election: true
I20260928 02:17:29.904049 6294 raft_consensus.cc:2781] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:17:29.904175 6294 raft_consensus.cc:482] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:17:29.904276 6294 raft_consensus.cc:3037] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:29.904193 5940 raft_consensus.cc:2454] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfc4b10f2ef44600bf1d82622687afaf in term 0.
I20260928 02:17:29.906847 6294 raft_consensus.cc:504] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } }
I20260928 02:17:29.907605 6294 leader_election.cc:288] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [CANDIDATE]: Term 1 election: Requested vote from peers 77bb4a4d5cb940ffa13f898b815371bf (127.4.140.193:41741), 414aa59e2e8f40e78885ff00831c4614 (127.4.140.195:34197)
I20260928 02:17:29.908120 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5c40cdc3cf42528c72575bfdc69e7c" candidate_uuid: "bfc4b10f2ef44600bf1d82622687afaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bb4a4d5cb940ffa13f898b815371bf"
I20260928 02:17:29.908169 6218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5c40cdc3cf42528c72575bfdc69e7c" candidate_uuid: "bfc4b10f2ef44600bf1d82622687afaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "414aa59e2e8f40e78885ff00831c4614"
I20260928 02:17:29.908367 5940 raft_consensus.cc:3037] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:29.908438 6218 raft_consensus.cc:3037] T 7b5c40cdc3cf42528c72575bfdc69e7c P 414aa59e2e8f40e78885ff00831c4614 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:29.911279 5940 raft_consensus.cc:2454] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfc4b10f2ef44600bf1d82622687afaf in term 1.
I20260928 02:17:29.911767 6012 leader_election.cc:302] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bb4a4d5cb940ffa13f898b815371bf, bfc4b10f2ef44600bf1d82622687afaf; no voters:
I20260928 02:17:29.911795 6218 raft_consensus.cc:2454] T 7b5c40cdc3cf42528c72575bfdc69e7c P 414aa59e2e8f40e78885ff00831c4614 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfc4b10f2ef44600bf1d82622687afaf in term 1.
I20260928 02:17:29.912159 6294 raft_consensus.cc:2781] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:29.912642 6294 raft_consensus.cc:686] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [term 1 LEADER]: Becoming Leader. State: Replica: bfc4b10f2ef44600bf1d82622687afaf, State: Running, Role: LEADER
I20260928 02:17:29.913091 6294 consensus_queue.cc:237] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } }
I20260928 02:17:29.916985 5781 catalog_manager.cc:5495] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf reported cstate change: term changed from 0 to 1, leader changed from <none> to bfc4b10f2ef44600bf1d82622687afaf (127.4.140.194). New cstate: current_term: 1 leader_uuid: "bfc4b10f2ef44600bf1d82622687afaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:17:29 dist-test-slave-h37r krb5kdc[5738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561847, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:29.970487 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.954899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56328 (local address 127.4.140.254:43303)
0928 02:17:29.955110 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:29.955118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:29.955766 (+ 648us) server_negotiation.cc:407] Connection header received
0928 02:17:29.956014 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:29.956019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:29.956099 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:29.956203 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:29.957389 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.958103 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.959343 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.959741 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.959928 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.964342 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:29.964348 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:29.964352 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:29.964380 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:29.967256 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:29.968233 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:29.968237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:29.968239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:29.968400 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:29.968868 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:29.968871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:29.968873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:29.969227 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:29.969438 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:29.969832 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 02:17:29.970182 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":159,"server-negotiator.queue_time_us":79}
I20260928 02:17:29.970521 6326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:29.954932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43303 (local address 127.0.0.1:56328)
0928 02:17:29.955650 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:29.955671 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:29.955848 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:29.956225 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:29.956228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:29.956931 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:29.957237 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:29.957246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.958230 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.958238 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:29.959202 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:29.959213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:29.960029 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:29.960037 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:29.960256 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:29.961012 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:29.961038 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:29.964110 (+ 3072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:29.967417 (+ 3307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:29.967431 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:29.967439 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:29.968097 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:29.968514 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:29.968520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:29.968525 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:29.968721 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:29.969429 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:29.969437 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:29.969595 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 02:17:29.969750 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 02:17:29.970061 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":1,"thread_start_us":416,"threads_started":1}
I20260928 02:17:29.976676 5781 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:56328:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:17:29.977748 5781 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:17:29.984508 6198 tablet_service.cc:1467] Processing CreateTablet for tablet 0709d2803ef24a8090320f3bc9f42c1a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6a6c1495c7ae437a8cd65c92bfb1ec60]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:17:29.984541 6059 tablet_service.cc:1467] Processing CreateTablet for tablet 0709d2803ef24a8090320f3bc9f42c1a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6a6c1495c7ae437a8cd65c92bfb1ec60]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:17:29.985028 6198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0709d2803ef24a8090320f3bc9f42c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:29.985110 6059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0709d2803ef24a8090320f3bc9f42c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:29.988686 5920 tablet_service.cc:1467] Processing CreateTablet for tablet 0709d2803ef24a8090320f3bc9f42c1a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6a6c1495c7ae437a8cd65c92bfb1ec60]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:17:29.989269 5920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0709d2803ef24a8090320f3bc9f42c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:29.990387 6289 tablet_bootstrap.cc:492] T 0709d2803ef24a8090320f3bc9f42c1a P 414aa59e2e8f40e78885ff00831c4614: Bootstrap starting.
I20260928 02:17:29.992084 6289 tablet_bootstrap.cc:654] T 0709d2803ef24a8090320f3bc9f42c1a P 414aa59e2e8f40e78885ff00831c4614: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:29.992689 6288 tablet_bootstrap.cc:492] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf: Bootstrap starting.
I20260928 02:17:29.994417 6288 tablet_bootstrap.cc:654] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:29.997895 6287 tablet_bootstrap.cc:492] T 0709d2803ef24a8090320f3bc9f42c1a P 77bb4a4d5cb940ffa13f898b815371bf: Bootstrap starting.
I20260928 02:17:30.000221 6287 tablet_bootstrap.cc:654] T 0709d2803ef24a8090320f3bc9f42c1a P 77bb4a4d5cb940ffa13f898b815371bf: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:30.004457 6287 tablet_bootstrap.cc:492] T 0709d2803ef24a8090320f3bc9f42c1a P 77bb4a4d5cb940ffa13f898b815371bf: No bootstrap required, opened a new log
I20260928 02:17:30.004771 6287 ts_tablet_manager.cc:1397] T 0709d2803ef24a8090320f3bc9f42c1a P 77bb4a4d5cb940ffa13f898b815371bf: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 02:17:30.005853 6287 raft_consensus.cc:348] T 0709d2803ef24a8090320f3bc9f42c1a P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } }
I20260928 02:17:30.006352 6287 raft_consensus.cc:374] T 0709d2803ef24a8090320f3bc9f42c1a P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:30.006502 6287 raft_consensus.cc:729] T 0709d2803ef24a8090320f3bc9f42c1a P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bb4a4d5cb940ffa13f898b815371bf, State: Initialized, Role: FOLLOWER
I20260928 02:17:30.007059 6287 consensus_queue.cc:260] T 0709d2803ef24a8090320f3bc9f42c1a P 77bb4a4d5cb940ffa13f898b815371bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } }
I20260928 02:17:30.007688 6298 tablet_replica.cc:439] T 0709d2803ef24a8090320f3bc9f42c1a P 77bb4a4d5cb940ffa13f898b815371bf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } }
I20260928 02:17:30.008119 6298 tablet_replica.cc:442] T 0709d2803ef24a8090320f3bc9f42c1a P 77bb4a4d5cb940ffa13f898b815371bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:17:30.008867 6287 ts_tablet_manager.cc:1428] T 0709d2803ef24a8090320f3bc9f42c1a P 77bb4a4d5cb940ffa13f898b815371bf: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 02:17:30.011009 6288 tablet_bootstrap.cc:492] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf: No bootstrap required, opened a new log
I20260928 02:17:30.011227 6288 ts_tablet_manager.cc:1397] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.004s
I20260928 02:17:30.012013 6288 raft_consensus.cc:348] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } }
I20260928 02:17:30.012315 6288 raft_consensus.cc:374] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:30.012431 6288 raft_consensus.cc:729] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfc4b10f2ef44600bf1d82622687afaf, State: Initialized, Role: FOLLOWER
I20260928 02:17:30.012737 6288 consensus_queue.cc:260] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } }
I20260928 02:17:30.013321 6288 ts_tablet_manager.cc:1428] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:17:30.013244 6294 tablet_replica.cc:439] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } }
I20260928 02:17:30.013564 6294 tablet_replica.cc:442] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:17:30.017381 6289 tablet_bootstrap.cc:492] T 0709d2803ef24a8090320f3bc9f42c1a P 414aa59e2e8f40e78885ff00831c4614: No bootstrap required, opened a new log
I20260928 02:17:30.017560 6289 ts_tablet_manager.cc:1397] T 0709d2803ef24a8090320f3bc9f42c1a P 414aa59e2e8f40e78885ff00831c4614: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.015s
I20260928 02:17:30.018237 6289 raft_consensus.cc:348] T 0709d2803ef24a8090320f3bc9f42c1a P 414aa59e2e8f40e78885ff00831c4614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } }
I20260928 02:17:30.018472 6289 raft_consensus.cc:374] T 0709d2803ef24a8090320f3bc9f42c1a P 414aa59e2e8f40e78885ff00831c4614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:30.018577 6289 raft_consensus.cc:729] T 0709d2803ef24a8090320f3bc9f42c1a P 414aa59e2e8f40e78885ff00831c4614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 414aa59e2e8f40e78885ff00831c4614, State: Initialized, Role: FOLLOWER
I20260928 02:17:30.018834 6289 consensus_queue.cc:260] T 0709d2803ef24a8090320f3bc9f42c1a P 414aa59e2e8f40e78885ff00831c4614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } }
I20260928 02:17:30.019228 6295 tablet_replica.cc:439] T 0709d2803ef24a8090320f3bc9f42c1a P 414aa59e2e8f40e78885ff00831c4614: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } }
I20260928 02:17:30.019531 6295 tablet_replica.cc:442] T 0709d2803ef24a8090320f3bc9f42c1a P 414aa59e2e8f40e78885ff00831c4614: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:17:30.020001 6289 ts_tablet_manager.cc:1428] T 0709d2803ef24a8090320f3bc9f42c1a P 414aa59e2e8f40e78885ff00831c4614: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260928 02:17:30.025933 6269 tablet.cc:2369] T 0709d2803ef24a8090320f3bc9f42c1a P 414aa59e2e8f40e78885ff00831c4614: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:17:30.081044 6294 raft_consensus.cc:482] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:17:30.081295 6294 raft_consensus.cc:504] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } }
I20260928 02:17:30.082052 6294 leader_election.cc:288] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 414aa59e2e8f40e78885ff00831c4614 (127.4.140.195:34197), 77bb4a4d5cb940ffa13f898b815371bf (127.4.140.193:41741)
I20260928 02:17:30.082692 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0709d2803ef24a8090320f3bc9f42c1a" candidate_uuid: "bfc4b10f2ef44600bf1d82622687afaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bb4a4d5cb940ffa13f898b815371bf" is_pre_election: true
I20260928 02:17:30.082751 6218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0709d2803ef24a8090320f3bc9f42c1a" candidate_uuid: "bfc4b10f2ef44600bf1d82622687afaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "414aa59e2e8f40e78885ff00831c4614" is_pre_election: true
I20260928 02:17:30.082975 6218 raft_consensus.cc:2454] T 0709d2803ef24a8090320f3bc9f42c1a P 414aa59e2e8f40e78885ff00831c4614 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfc4b10f2ef44600bf1d82622687afaf in term 0.
I20260928 02:17:30.083021 5940 raft_consensus.cc:2454] T 0709d2803ef24a8090320f3bc9f42c1a P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfc4b10f2ef44600bf1d82622687afaf in term 0.
I20260928 02:17:30.083395 6012 leader_election.cc:302] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bb4a4d5cb940ffa13f898b815371bf, bfc4b10f2ef44600bf1d82622687afaf; no voters:
I20260928 02:17:30.083684 6294 raft_consensus.cc:2781] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:17:30.083792 6294 raft_consensus.cc:482] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:17:30.083936 6294 raft_consensus.cc:3037] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:30.086982 6294 raft_consensus.cc:504] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } }
I20260928 02:17:30.087637 6294 leader_election.cc:288] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf [CANDIDATE]: Term 1 election: Requested vote from peers 414aa59e2e8f40e78885ff00831c4614 (127.4.140.195:34197), 77bb4a4d5cb940ffa13f898b815371bf (127.4.140.193:41741)
I20260928 02:17:30.088135 6218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0709d2803ef24a8090320f3bc9f42c1a" candidate_uuid: "bfc4b10f2ef44600bf1d82622687afaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "414aa59e2e8f40e78885ff00831c4614"
I20260928 02:17:30.088352 6218 raft_consensus.cc:3037] T 0709d2803ef24a8090320f3bc9f42c1a P 414aa59e2e8f40e78885ff00831c4614 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:30.091185 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0709d2803ef24a8090320f3bc9f42c1a" candidate_uuid: "bfc4b10f2ef44600bf1d82622687afaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bb4a4d5cb940ffa13f898b815371bf"
I20260928 02:17:30.091297 6218 raft_consensus.cc:2454] T 0709d2803ef24a8090320f3bc9f42c1a P 414aa59e2e8f40e78885ff00831c4614 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfc4b10f2ef44600bf1d82622687afaf in term 1.
I20260928 02:17:30.091432 5940 raft_consensus.cc:3037] T 0709d2803ef24a8090320f3bc9f42c1a P 77bb4a4d5cb940ffa13f898b815371bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:30.091753 6011 leader_election.cc:302] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 414aa59e2e8f40e78885ff00831c4614, bfc4b10f2ef44600bf1d82622687afaf; no voters:
I20260928 02:17:30.092124 6294 raft_consensus.cc:2781] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:30.092345 6294 raft_consensus.cc:686] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf [term 1 LEADER]: Becoming Leader. State: Replica: bfc4b10f2ef44600bf1d82622687afaf, State: Running, Role: LEADER
I20260928 02:17:30.092942 6294 consensus_queue.cc:237] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } }
I20260928 02:17:30.094161 5940 raft_consensus.cc:2454] T 0709d2803ef24a8090320f3bc9f42c1a P 77bb4a4d5cb940ffa13f898b815371bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfc4b10f2ef44600bf1d82622687afaf in term 1.
I20260928 02:17:30.094715 6319 tablet_replica.cc:439] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf: TxnStatusTablet state changed. Reason: New leader bfc4b10f2ef44600bf1d82622687afaf. Latest consensus state: current_term: 1 leader_uuid: "bfc4b10f2ef44600bf1d82622687afaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } }
I20260928 02:17:30.095194 6319 tablet_replica.cc:442] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:17:30.096063 6331 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:17:30.096282 6331 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:17:30.096632 5781 catalog_manager.cc:5495] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf reported cstate change: term changed from 0 to 1, leader changed from <none> to bfc4b10f2ef44600bf1d82622687afaf (127.4.140.194). New cstate: current_term: 1 leader_uuid: "bfc4b10f2ef44600bf1d82622687afaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } health_report { overall_health: UNKNOWN } } }
I20260928 02:17:30.128286 6326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:30.120480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:43935 (local address 127.0.0.1:50274)
0928 02:17:30.120835 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:30.120868 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:30.121012 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:30.121587 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:30.121592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:30.121607 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:17:30.121807 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:30.121818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:30.123481 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:30.123487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:30.125099 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:30.125114 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:30.126182 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:30.126190 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:30.126409 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:30.126486 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:17:30.127668 (+ 1182us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:17:30.127697 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:17:30.127794 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 02:17:30.127974 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260928 02:17:30.128338 6286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:30.120490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50274 (local address 127.4.140.194:43935)
0928 02:17:30.120699 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:30.120707 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:30.120955 (+ 248us) server_negotiation.cc:407] Connection header received
0928 02:17:30.121250 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:30.121258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:30.121379 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:30.121548 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:17:30.121962 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:30.123330 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:30.125274 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:30.126009 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:30.126199 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:30.126697 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:17:30.127541 (+ 844us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:17:30.127702 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:30.127836 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 02:17:30.127964 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:17:30.133671 5940 raft_consensus.cc:1261] T 0709d2803ef24a8090320f3bc9f42c1a P 77bb4a4d5cb940ffa13f898b815371bf [term 1 FOLLOWER]: Refusing update from remote peer bfc4b10f2ef44600bf1d82622687afaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:17:30.133895 6218 raft_consensus.cc:1261] T 0709d2803ef24a8090320f3bc9f42c1a P 414aa59e2e8f40e78885ff00831c4614 [term 1 FOLLOWER]: Refusing update from remote peer bfc4b10f2ef44600bf1d82622687afaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:17:30.134657 6294 consensus_queue.cc:1035] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:17:30.135103 6319 consensus_queue.cc:1035] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:17:30.138638 6295 tablet_replica.cc:439] T 0709d2803ef24a8090320f3bc9f42c1a P 414aa59e2e8f40e78885ff00831c4614: TxnStatusTablet state changed. Reason: New leader bfc4b10f2ef44600bf1d82622687afaf. Latest consensus state: current_term: 1 leader_uuid: "bfc4b10f2ef44600bf1d82622687afaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } }
I20260928 02:17:30.138922 6295 tablet_replica.cc:442] T 0709d2803ef24a8090320f3bc9f42c1a P 414aa59e2e8f40e78885ff00831c4614: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:17:30.141790 6294 tablet_replica.cc:439] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bfc4b10f2ef44600bf1d82622687afaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } }
I20260928 02:17:30.142086 6294 tablet_replica.cc:442] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:17:30.144407 6302 mvcc.cc:205] Tried to move back new op lower bound from 7334141338138419200 to 7334141337983549440. Current Snapshot: MvccSnapshot[applied={T|T < 7334141338138419200 or (T in {7334141338138419200})}]
I20260928 02:17:30.145138 6295 tablet_replica.cc:439] T 0709d2803ef24a8090320f3bc9f42c1a P 414aa59e2e8f40e78885ff00831c4614: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bfc4b10f2ef44600bf1d82622687afaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } }
I20260928 02:17:30.145385 6295 tablet_replica.cc:442] T 0709d2803ef24a8090320f3bc9f42c1a P 414aa59e2e8f40e78885ff00831c4614: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:17:30.157720 6298 tablet_replica.cc:439] T 0709d2803ef24a8090320f3bc9f42c1a P 77bb4a4d5cb940ffa13f898b815371bf: TxnStatusTablet state changed. Reason: New leader bfc4b10f2ef44600bf1d82622687afaf. Latest consensus state: current_term: 1 leader_uuid: "bfc4b10f2ef44600bf1d82622687afaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } }
I20260928 02:17:30.158107 6298 tablet_replica.cc:442] T 0709d2803ef24a8090320f3bc9f42c1a P 77bb4a4d5cb940ffa13f898b815371bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:17:30.160403 6319 tablet_replica.cc:439] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bfc4b10f2ef44600bf1d82622687afaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } }
I20260928 02:17:30.160732 6319 tablet_replica.cc:442] T 0709d2803ef24a8090320f3bc9f42c1a P bfc4b10f2ef44600bf1d82622687afaf: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:17:30.162079 6286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:30.154866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50286 (local address 127.4.140.194:43935)
0928 02:17:30.155297 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:30.155304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:30.155323 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:17:30.155389 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:30.155395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:30.155478 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:30.156086 (+ 608us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:17:30.156242 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:30.157166 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:30.159202 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:30.159937 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:30.160648 (+ 711us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:30.160783 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:17:30.160925 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:17:30.161623 (+ 698us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:30.161679 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 02:17:30.161767 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281}
I20260928 02:17:30.166347 6298 tablet_replica.cc:439] T 0709d2803ef24a8090320f3bc9f42c1a P 77bb4a4d5cb940ffa13f898b815371bf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bfc4b10f2ef44600bf1d82622687afaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc4b10f2ef44600bf1d82622687afaf" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43935 } } peers { permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 } } peers { permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 } } }
I20260928 02:17:30.166656 6298 tablet_replica.cc:442] T 0709d2803ef24a8090320f3bc9f42c1a P 77bb4a4d5cb940ffa13f898b815371bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:17:30.183686 6286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:30.175438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:47271 (local address 127.4.140.194:43935)
0928 02:17:30.175644 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:30.175650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:30.176445 (+ 795us) server_negotiation.cc:407] Connection header received
0928 02:17:30.176669 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:30.176674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:30.176767 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:30.176869 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:17:30.177557 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:30.178880 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:30.181139 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:30.181916 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:30.182082 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:30.182659 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:17:30.182837 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:17:30.182935 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:30.183092 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 02:17:30.183236 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":58}
I20260928 02:17:30.183743 6338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:30.175516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:43935 (local address 127.4.140.194:47271)
0928 02:17:30.176317 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:30.176347 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:30.176512 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:30.176912 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:30.176918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:30.176931 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:17:30.177351 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:30.177367 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:30.179032 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:30.179037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:30.180979 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:30.180995 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:30.182143 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:30.182150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:30.182346 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:30.182474 (+ 128us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:17:30.182956 (+ 482us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:17:30.182967 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 02:17:30.183057 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 02:17:30.183214 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":458,"threads_started":1}
I20260928 02:17:30.196501 6218 raft_consensus.cc:1261] T 7b5c40cdc3cf42528c72575bfdc69e7c P 414aa59e2e8f40e78885ff00831c4614 [term 1 FOLLOWER]: Refusing update from remote peer bfc4b10f2ef44600bf1d82622687afaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:17:30.196605 5940 raft_consensus.cc:1261] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf [term 1 FOLLOWER]: Refusing update from remote peer bfc4b10f2ef44600bf1d82622687afaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:17:30.197134 6319 consensus_queue.cc:1035] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bb4a4d5cb940ffa13f898b815371bf" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:17:30.197400 6294 consensus_queue.cc:1035] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "414aa59e2e8f40e78885ff00831c4614" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 34197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:17:30.291558 5782 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:17:30.291811 5782 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:17:30.294770 5782 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5656ff10a184535a121fabbd68fb762: Sending DeleteTablet for 3 replicas of tablet 7b5c40cdc3cf42528c72575bfdc69e7c
I20260928 02:17:30.295563 5920 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b5c40cdc3cf42528c72575bfdc69e7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:30 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:53388
I20260928 02:17:30.295930 6312 tablet_replica.cc:330] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf: stopping tablet replica
I20260928 02:17:30.296140 6312 raft_consensus.cc:2229] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:17:30.296306 6198 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b5c40cdc3cf42528c72575bfdc69e7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:30 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:51038
I20260928 02:17:30.296357 6312 raft_consensus.cc:2258] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:30.296578 6313 tablet_replica.cc:330] T 7b5c40cdc3cf42528c72575bfdc69e7c P 414aa59e2e8f40e78885ff00831c4614: stopping tablet replica
I20260928 02:17:30.296793 6313 raft_consensus.cc:2229] T 7b5c40cdc3cf42528c72575bfdc69e7c P 414aa59e2e8f40e78885ff00831c4614 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:17:30.296993 6313 raft_consensus.cc:2258] T 7b5c40cdc3cf42528c72575bfdc69e7c P 414aa59e2e8f40e78885ff00831c4614 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:30.297417 6059 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b5c40cdc3cf42528c72575bfdc69e7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:30 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:50262
I20260928 02:17:30.298352 6311 tablet_replica.cc:330] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf: stopping tablet replica
I20260928 02:17:30.298342 6313 ts_tablet_manager.cc:1905] T 7b5c40cdc3cf42528c72575bfdc69e7c P 414aa59e2e8f40e78885ff00831c4614: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:30.298537 6311 raft_consensus.cc:2229] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:17:30.299072 6311 raft_consensus.cc:2258] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:30.300338 6311 ts_tablet_manager.cc:1905] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:30.305406 6313 ts_tablet_manager.cc:1918] T 7b5c40cdc3cf42528c72575bfdc69e7c P 414aa59e2e8f40e78885ff00831c4614: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:17:30.305615 6313 log.cc:1196] T 7b5c40cdc3cf42528c72575bfdc69e7c P 414aa59e2e8f40e78885ff00831c4614: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/7b5c40cdc3cf42528c72575bfdc69e7c
I20260928 02:17:30.306092 6313 ts_tablet_manager.cc:1939] T 7b5c40cdc3cf42528c72575bfdc69e7c P 414aa59e2e8f40e78885ff00831c4614: Deleting consensus metadata
I20260928 02:17:30.307119 6312 ts_tablet_manager.cc:1905] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:30.307704 6311 ts_tablet_manager.cc:1918] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:17:30.307863 6311 log.cc:1196] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/7b5c40cdc3cf42528c72575bfdc69e7c
I20260928 02:17:30.308329 6311 ts_tablet_manager.cc:1939] T 7b5c40cdc3cf42528c72575bfdc69e7c P bfc4b10f2ef44600bf1d82622687afaf: Deleting consensus metadata
I20260928 02:17:30.309458 5767 catalog_manager.cc:4841] TS bfc4b10f2ef44600bf1d82622687afaf (127.4.140.194:43935): tablet 7b5c40cdc3cf42528c72575bfdc69e7c (table test-table [id=681384452b7f4bb0a1ebe0a9de993395]) successfully deleted
I20260928 02:17:30.313508 5768 catalog_manager.cc:4841] TS 414aa59e2e8f40e78885ff00831c4614 (127.4.140.195:34197): tablet 7b5c40cdc3cf42528c72575bfdc69e7c (table test-table [id=681384452b7f4bb0a1ebe0a9de993395]) successfully deleted
I20260928 02:17:30.316538 6312 ts_tablet_manager.cc:1918] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:17:30.316850 6312 log.cc:1196] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/7b5c40cdc3cf42528c72575bfdc69e7c
I20260928 02:17:30.317447 6312 ts_tablet_manager.cc:1939] T 7b5c40cdc3cf42528c72575bfdc69e7c P 77bb4a4d5cb940ffa13f898b815371bf: Deleting consensus metadata
I20260928 02:17:30.318583 5769 catalog_manager.cc:4841] TS 77bb4a4d5cb940ffa13f898b815371bf (127.4.140.193:41741): tablet 7b5c40cdc3cf42528c72575bfdc69e7c (table test-table [id=681384452b7f4bb0a1ebe0a9de993395]) successfully deleted
Sep 28 02:17:30 dist-test-slave-h37r krb5kdc[5738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561849, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:17:30.334652 6281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:30.318618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53406 (local address 127.4.140.193:41741)
0928 02:17:30.318832 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:30.318838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:30.319869 (+ 1031us) server_negotiation.cc:407] Connection header received
0928 02:17:30.320110 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:30.320115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:30.320195 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:30.320312 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:30.321809 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:30.322550 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:30.323862 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:30.324344 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:30.324487 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:30.328399 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:30.328412 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:30.328429 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:30.328460 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:30.331774 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:30.332491 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:30.332501 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:30.332504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:30.332632 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:30.333014 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:30.333019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:30.333029 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:30.333407 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:30.333744 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:30.334026 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 02:17:30.334247 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 02:17:30.335704 5985 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53406
I20260928 02:17:30.352201 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:30.339207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.4.140.254:43303)
0928 02:17:30.339382 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:30.339388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:30.339988 (+ 600us) server_negotiation.cc:407] Connection header received
0928 02:17:30.340196 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:30.340202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:30.340292 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:30.340398 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:30.341760 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:30.342552 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:30.343843 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:30.344340 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:30.344482 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:30.346762 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:30.346769 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:30.346773 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:30.346817 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:30.349541 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:30.350176 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:30.350183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:30.350192 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:30.350321 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:30.350712 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:30.350718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:30.350721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:30.350986 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:30.351271 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:30.351566 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 02:17:30.351782 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 02:17:30.352789 5782 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56342
I20260928 02:17:30.354856 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 5854
I20260928 02:17:30.378621 6281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:30.356065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53416 (local address 127.4.140.193:41741)
0928 02:17:30.356271 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:30.356277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:30.365581 (+ 9304us) server_negotiation.cc:407] Connection header received
0928 02:17:30.365797 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:30.365802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:30.365886 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:30.365998 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:30.367330 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:30.368427 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:30.369783 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:30.370255 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:30.370419 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:30.372713 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:30.372719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:30.372725 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:30.372764 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:30.375829 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:30.376565 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:30.376571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:30.376574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:30.376727 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:30.377102 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:30.377108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:30.377111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:30.377413 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:30.377659 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:30.377975 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 02:17:30.378203 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 02:17:30.379156 5985 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53416
W20260928 02:17:30.379760 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 5854: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:17:30.379967 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 5854
I20260928 02:17:30.404896 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 5994
Sep 28 02:17:30 dist-test-slave-h37r krb5kdc[5738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561849, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.140.194@KRBTEST.COM
I20260928 02:17:30.421001 6286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:30.405716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50288 (local address 127.4.140.194:43935)
0928 02:17:30.405896 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:30.405902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:30.405923 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:17:30.406090 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:30.406101 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:30.406212 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:30.406333 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:30.407406 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:30.408573 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:30.410243 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:30.410857 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:30.410991 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:30.415101 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:30.415113 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:30.415129 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:30.415177 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:30.418196 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:30.418946 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:30.418953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:30.418955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:30.419070 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:30.419472 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:30.419475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:30.419477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:30.419846 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:30.420191 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:30.420530 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 02:17:30.420688 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 02:17:30.421744 6124 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50288
W20260928 02:17:30.422425 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 5994: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:17:30.422634 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 5994
I20260928 02:17:30.438328 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 6133
Sep 28 02:17:30 dist-test-slave-h37r krb5kdc[5738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561849, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.140.195@KRBTEST.COM
I20260928 02:17:30.454353 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:30.439110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.4.140.195:34197)
0928 02:17:30.439649 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:30.439653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:30.439669 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:17:30.439721 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:30.439725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:30.439785 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:30.439892 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:30.441115 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:30.442024 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:30.443407 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:30.443831 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:30.444084 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:30.448370 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:30.448385 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:30.448402 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:30.448439 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:30.451390 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:30.452196 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:30.452204 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:30.452207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:30.452367 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:30.452764 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:30.452768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:30.452770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:30.453151 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:30.453483 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:30.453797 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 02:17:30.454021 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":311,"threads_started":1}
W20260928 02:17:30.455044 6263 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51048
W20260928 02:17:30.455695 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 6133: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:17:30.455865 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 6133
I20260928 02:17:30.473253 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 5751
I20260928 02:17:30.474534 5823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:30.589665 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 5751
2026-09-28T02:17:30Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (7135 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:17:30 dist-test-slave-h37r krb5kdc[6353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:17:30 dist-test-slave-h37r krb5kdc[6353](info): set up 2 sockets
Sep 28 02:17:30 dist-test-slave-h37r krb5kdc[6353](info): commencing operation
krb5kdc: starting...
W20260928 02:17:31.684089 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:17:31 dist-test-slave-h37r krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:17:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:17:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:17:31.875567 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:34297
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:44425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:34297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:32.203118 6366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:32.203421 6366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:32.203593 6366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:32.211206 6366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:17:32.211347 6366 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:17:32.211445 6366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:32.211567 6366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:32.211678 6366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:17:32.211802 6366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:17:32.222470 6366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:44425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:34297
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:34297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:32.224036 6366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:32.225483 6366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:32.239683 6371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:32.239713 6372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:32.239738 6374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:32.241016 6366 server_base.cc:1034] running on GCE node
I20260928 02:17:32.242027 6366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:32.243599 6366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:32.244856 6366 hybrid_clock.cc:648] HybridClock initialized: now 1790561852244760 us; error 70 us; skew 500 ppm
I20260928 02:17:32.245203 6366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:32 dist-test-slave-h37r krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561852, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:32.248575 6366 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:17:32.250356 6366 webserver.cc:460] Webserver started at http://127.4.140.254:35745/ using document root <none> and password file <none>
I20260928 02:17:32.251125 6366 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:32.251284 6366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:32.251636 6366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:32.255134 6366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "b6c786e2fd86435ebfd299ea6bed2b7e"
format_stamp: "Formatted at 2026-09-28 02:17:32 on dist-test-slave-h37r"
server_key: "cd741c9c2f0b28dbd28e479757afc92c"
server_key_iv: "8a50dd2669f3fefa0320f88b7aba7b3a"
server_key_version: "encryptionkey@0"
I20260928 02:17:32.256089 6366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "b6c786e2fd86435ebfd299ea6bed2b7e"
format_stamp: "Formatted at 2026-09-28 02:17:32 on dist-test-slave-h37r"
server_key: "cd741c9c2f0b28dbd28e479757afc92c"
server_key_iv: "8a50dd2669f3fefa0320f88b7aba7b3a"
server_key_version: "encryptionkey@0"
I20260928 02:17:32.261900 6366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:17:32.265898 6381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:32.267376 6366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:17:32.267583 6366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "b6c786e2fd86435ebfd299ea6bed2b7e"
format_stamp: "Formatted at 2026-09-28 02:17:32 on dist-test-slave-h37r"
server_key: "cd741c9c2f0b28dbd28e479757afc92c"
server_key_iv: "8a50dd2669f3fefa0320f88b7aba7b3a"
server_key_version: "encryptionkey@0"
I20260928 02:17:32.267827 6366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:32.293032 6366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:32.297819 6366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:32.298108 6366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:32.321925 6366 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:34297
I20260928 02:17:32.321955 6442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:34297 every 8 connection(s)
I20260928 02:17:32.323398 6366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:17:32.325285 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 6366
I20260928 02:17:32.325531 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:17:32.326575 4659 external_mini_cluster.cc:1428] Setting key e75e36b6052102f1f8a46dbd7d85e306
I20260928 02:17:32.329272 6443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 02:17:32 dist-test-slave-h37r krb5kdc[6353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:32.345315 6443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e: Bootstrap starting.
I20260928 02:17:32.350446 6443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:32.351991 6443 log.cc:824] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:32.355485 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:32.329030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40998 (local address 127.4.140.254:34297)
0928 02:17:32.329883 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:32.329896 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:32.329938 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:17:32.331686 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:32.331708 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:32.332196 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:32.332564 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:32.334725 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:32.336249 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:32.338414 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:32.339623 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:32.339837 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:32.345362 (+ 5525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:32.345377 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:32.345403 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:32.345473 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:32.350115 (+ 4642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:32.351072 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:32.351082 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:32.351092 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:32.351335 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:32.351764 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:32.351770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:32.351773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:32.352741 (+ 968us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:32.353078 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:32.353630 (+ 552us) server_negotiation.cc:299] Negotiation successful
0928 02:17:32.354098 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":410,"threads_started":1}
I20260928 02:17:32.355825 6443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e: No bootstrap required, opened a new log
I20260928 02:17:32.364631 6443 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c786e2fd86435ebfd299ea6bed2b7e" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 34297 } }
I20260928 02:17:32.365051 6443 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:32.365175 6443 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c786e2fd86435ebfd299ea6bed2b7e, State: Initialized, Role: FOLLOWER
I20260928 02:17:32.365869 6443 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c786e2fd86435ebfd299ea6bed2b7e" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 34297 } }
I20260928 02:17:32.366083 6443 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:17:32.366214 6443 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:17:32.366402 6443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:32.434540 6443 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c786e2fd86435ebfd299ea6bed2b7e" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 34297 } }
I20260928 02:17:32.435245 6443 leader_election.cc:302] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6c786e2fd86435ebfd299ea6bed2b7e; no voters:
I20260928 02:17:32.435948 6443 leader_election.cc:288] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:17:32.436110 6448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:32.438313 6443 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:17:32.438787 6448 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e [term 1 LEADER]: Becoming Leader. State: Replica: b6c786e2fd86435ebfd299ea6bed2b7e, State: Running, Role: LEADER
I20260928 02:17:32.439363 6448 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c786e2fd86435ebfd299ea6bed2b7e" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 34297 } }
I20260928 02:17:32.442656 6449 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6c786e2fd86435ebfd299ea6bed2b7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c786e2fd86435ebfd299ea6bed2b7e" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 34297 } } }
I20260928 02:17:32.443102 6449 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:32.443558 6450 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6c786e2fd86435ebfd299ea6bed2b7e. Latest consensus state: current_term: 1 leader_uuid: "b6c786e2fd86435ebfd299ea6bed2b7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c786e2fd86435ebfd299ea6bed2b7e" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 34297 } } }
I20260928 02:17:32.443887 6450 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:32.444096 6460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:17:32.448743 6460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:17:32.457316 6460 catalog_manager.cc:1347] Generated new cluster ID: b89b43e4aaa44b6586e51579dafb712c
I20260928 02:17:32.457445 6460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:17:32.471982 6460 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:17:32.474028 6460 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:17:32.493441 6460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e: Generated new TSK 0
I20260928 02:17:32.494390 6460 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
W20260928 02:17:32.792115 4659 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.4.140.193: real 0.291s user 0.003s sys 0.008s
I20260928 02:17:32.793411 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:34297
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:44425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:33.124010 6469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:33.124351 6469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:33.124528 6469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:33.132020 6469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:33.132174 6469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:33.132340 6469 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:17:33.132427 6469 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:17:33.132535 6469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:17:33.143368 6469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:44425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.140.254:34297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:33.144769 6469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:33.146116 6469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:33.159150 6477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:33.159185 6474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:33.159219 6475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:33.160006 6469 server_base.cc:1034] running on GCE node
I20260928 02:17:33.160601 6469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:33.161612 6469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:33.162855 6469 hybrid_clock.cc:648] HybridClock initialized: now 1790561853162791 us; error 49 us; skew 500 ppm
I20260928 02:17:33.163192 6469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:33 dist-test-slave-h37r krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:33.174199 6469 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:17:33.175722 6469 webserver.cc:460] Webserver started at http://127.4.140.193:35999/ using document root <none> and password file <none>
I20260928 02:17:33.176448 6469 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:33.176556 6469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:33.176820 6469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:33.179582 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "554900af951f497489f6a9b68f810d55"
format_stamp: "Formatted at 2026-09-28 02:17:33 on dist-test-slave-h37r"
server_key: "5d5a24c839c5afd666537c07d82aaf28"
server_key_iv: "2f3ddd1a7517ca464e3cacc7b6120b91"
server_key_version: "encryptionkey@0"
I20260928 02:17:33.180328 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "554900af951f497489f6a9b68f810d55"
format_stamp: "Formatted at 2026-09-28 02:17:33 on dist-test-slave-h37r"
server_key: "5d5a24c839c5afd666537c07d82aaf28"
server_key_iv: "2f3ddd1a7517ca464e3cacc7b6120b91"
server_key_version: "encryptionkey@0"
I20260928 02:17:33.185320 6469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 02:17:33.188746 6484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:33.190178 6469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:17:33.190418 6469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "554900af951f497489f6a9b68f810d55"
format_stamp: "Formatted at 2026-09-28 02:17:33 on dist-test-slave-h37r"
server_key: "5d5a24c839c5afd666537c07d82aaf28"
server_key_iv: "2f3ddd1a7517ca464e3cacc7b6120b91"
server_key_version: "encryptionkey@0"
I20260928 02:17:33.190603 6469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:33.213533 6469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:33.217892 6469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:33.218204 6469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:33.221051 6469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:33.221171 6469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:33.221282 6469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:33.221390 6469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:17:33 dist-test-slave-h37r krb5kdc[6353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:33.246879 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:33.222388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:34297 (local address 127.4.140.193:48771)
0928 02:17:33.223231 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:33.223315 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:33.225205 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:33.225964 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:33.225973 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:33.226810 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:33.227733 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:33.227756 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:33.229186 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:33.229195 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:33.232338 (+ 3143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:33.232358 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:33.233268 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:33.233278 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:33.233633 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:33.234611 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:33.234686 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:33.238048 (+ 3362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:33.241602 (+ 3554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:33.241618 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:33.241636 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:33.242150 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:33.242603 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:33.242609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:33.242612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:33.242955 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:33.243738 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:33.243746 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:33.244326 (+ 580us) client_negotiation.cc:769] Sending connection context
0928 02:17:33.244663 (+ 337us) client_negotiation.cc:240] Negotiation successful
0928 02:17:33.245179 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":366,"threads_started":1}
I20260928 02:17:33.247786 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:33.221669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:48771 (local address 127.4.140.254:34297)
0928 02:17:33.222355 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:33.222359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:33.223438 (+ 1079us) server_negotiation.cc:407] Connection header received
0928 02:17:33.225495 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:33.225501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:33.225588 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:33.225714 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:33.227940 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:33.228949 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:33.232536 (+ 3587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:33.232998 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:33.233105 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:33.238274 (+ 5169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:33.238281 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:33.238287 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:33.238349 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:33.241445 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:33.242317 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:33.242332 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:33.242337 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:33.242501 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:33.243141 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:33.243149 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:33.243154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:33.243505 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:33.243660 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:33.247376 (+ 3716us) server_negotiation.cc:299] Negotiation successful
0928 02:17:33.247549 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":474,"threads_started":1}
I20260928 02:17:33.269505 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:33.253829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:47215 (local address 127.4.140.254:34297)
0928 02:17:33.254078 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:33.254086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:33.254604 (+ 518us) server_negotiation.cc:407] Connection header received
0928 02:17:33.254840 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:33.254845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:33.254913 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:33.255019 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:33.256596 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:33.257549 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:33.260068 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:33.260535 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:33.260630 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:33.263348 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:33.263355 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:33.263361 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:33.263405 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:33.266745 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:33.267443 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:33.267449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:33.267452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:33.267616 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:33.268006 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:33.268014 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:33.268017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:33.268363 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:33.268530 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:33.268930 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 02:17:33.269146 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 02:17:33.269609 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:33.253752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:34297 (local address 127.4.140.193:47215)
0928 02:17:33.254477 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:33.254502 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:33.254670 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:33.255077 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:33.255084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:33.255843 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:33.256396 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:33.256415 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:33.257701 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:33.257712 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:33.259910 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:33.259930 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:33.260738 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:33.260746 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:33.261019 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:33.261742 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:33.261774 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:33.263101 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:33.266878 (+ 3777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:33.266885 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:33.266889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:33.267307 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:33.267726 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:33.267731 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:33.267734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:33.267870 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:33.268483 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:33.268487 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:33.268675 (+ 188us) client_negotiation.cc:769] Sending connection context
0928 02:17:33.268816 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 02:17:33.269147 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":361,"threads_started":1}
I20260928 02:17:33.280957 6469 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:35461
I20260928 02:17:33.280987 6604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:35461 every 8 connection(s)
I20260928 02:17:33.282157 6469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:17:33.283812 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 6469
I20260928 02:17:33.284050 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:17:33.284761 4659 external_mini_cluster.cc:1428] Setting key 77700ee213ef85fc4c79562df2008502
I20260928 02:17:33.284375 6605 heartbeater.cc:347] Connected to a master server at 127.4.140.254:34297
I20260928 02:17:33.285959 6605 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:33.286695 6605 heartbeater.cc:511] Master 127.4.140.254:34297 requested a full tablet report, sending...
I20260928 02:17:33.289080 6398 ts_manager.cc:194] Registered new tserver with Master: 554900af951f497489f6a9b68f810d55 (127.4.140.193:35461)
I20260928 02:17:33.291224 6398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:48771
WARNING: no policy specified for kudu/127.4.140.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
I20260928 02:17:33.330041 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:34297
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:44425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:33.654721 6609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:33.655026 6609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:33.655247 6609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:33.662431 6609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:33.662541 6609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:33.662699 6609 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:17:33.662786 6609 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:17:33.662920 6609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:17:33.673034 6609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:44425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.140.254:34297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:33.674401 6609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:33.675786 6609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:33.689582 6614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:33.689615 6615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:33.690115 6617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:33.690965 6609 server_base.cc:1034] running on GCE node
I20260928 02:17:33.691640 6609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:33.692749 6609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:33.694025 6609 hybrid_clock.cc:648] HybridClock initialized: now 1790561853693974 us; error 54 us; skew 500 ppm
I20260928 02:17:33.694474 6609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:33 dist-test-slave-h37r krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:33.698153 6609 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:17:33.699707 6609 webserver.cc:460] Webserver started at http://127.4.140.194:43297/ using document root <none> and password file <none>
I20260928 02:17:33.700480 6609 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:33.700589 6609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:33.700868 6609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:33.703871 6609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "afc578d2b43a495382ccde49ffa32ad1"
format_stamp: "Formatted at 2026-09-28 02:17:33 on dist-test-slave-h37r"
server_key: "4de6a6b24377b61c14eb8e5d284c3142"
server_key_iv: "9f2731aa65d2e011ba613ed3dc400bcb"
server_key_version: "encryptionkey@0"
I20260928 02:17:33.704716 6609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "afc578d2b43a495382ccde49ffa32ad1"
format_stamp: "Formatted at 2026-09-28 02:17:33 on dist-test-slave-h37r"
server_key: "4de6a6b24377b61c14eb8e5d284c3142"
server_key_iv: "9f2731aa65d2e011ba613ed3dc400bcb"
server_key_version: "encryptionkey@0"
I20260928 02:17:33.710067 6609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 02:17:33.713726 6624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:33.715123 6609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:17:33.715305 6609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "afc578d2b43a495382ccde49ffa32ad1"
format_stamp: "Formatted at 2026-09-28 02:17:33 on dist-test-slave-h37r"
server_key: "4de6a6b24377b61c14eb8e5d284c3142"
server_key_iv: "9f2731aa65d2e011ba613ed3dc400bcb"
server_key_version: "encryptionkey@0"
I20260928 02:17:33.715497 6609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:33.742385 6609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:33.747087 6609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:33.747442 6609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:33.750774 6609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:33.750905 6609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:33.751026 6609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:33.751121 6609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:17:33 dist-test-slave-h37r krb5kdc[6353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:33.776162 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:33.751404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:48071 (local address 127.4.140.254:34297)
0928 02:17:33.751619 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:33.751626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:33.753349 (+ 1723us) server_negotiation.cc:407] Connection header received
0928 02:17:33.755558 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:33.755564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:33.755658 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:33.755771 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:33.758117 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:33.759451 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:33.762151 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:33.762772 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:33.762945 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:33.768497 (+ 5552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:33.768506 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:33.768513 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:33.768563 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:33.771662 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:33.772893 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:33.772902 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:33.772907 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:33.773081 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:33.773746 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:33.773751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:33.773756 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:33.774083 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:33.774205 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:33.775392 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0928 02:17:33.775676 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 02:17:33.778209 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:33.752284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:34297 (local address 127.4.140.194:48071)
0928 02:17:33.753168 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:33.753216 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:33.755265 (+ 2049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:33.756132 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:33.756148 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:33.756989 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:33.757881 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:33.757922 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:33.759654 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:33.759660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:33.761962 (+ 2302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:33.761974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:33.763297 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:33.763303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:33.763610 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:33.764565 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:33.764607 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:33.768174 (+ 3567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:33.771815 (+ 3641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:33.771836 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:33.771862 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:33.772729 (+ 867us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:33.773209 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:33.773217 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:33.773223 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:33.773586 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:33.774186 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:33.774195 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:33.774765 (+ 570us) client_negotiation.cc:769] Sending connection context
0928 02:17:33.775246 (+ 481us) client_negotiation.cc:240] Negotiation successful
0928 02:17:33.776044 (+ 798us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":362,"threads_started":1}
I20260928 02:17:33.800231 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:33.784909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:47597 (local address 127.4.140.254:34297)
0928 02:17:33.785095 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:33.785101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:33.785671 (+ 570us) server_negotiation.cc:407] Connection header received
0928 02:17:33.785879 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:33.785884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:33.785967 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:33.786070 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:33.786996 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:33.787722 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:33.789761 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:33.790527 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:33.790685 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:33.793427 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:33.793432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:33.793435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:33.793477 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:33.797075 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:33.798059 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:33.798066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:33.798071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:33.798206 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:33.798594 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:33.798601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:33.798607 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:33.798975 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:33.799141 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:33.799730 (+ 589us) server_negotiation.cc:299] Negotiation successful
0928 02:17:33.799994 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:17:33.801326 6705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:33.784968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:34297 (local address 127.4.140.194:47597)
0928 02:17:33.785565 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:33.785583 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:33.785743 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:33.786095 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:33.786099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:33.786576 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:33.786853 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:33.786864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:33.787860 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:33.787867 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:33.789556 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:33.789613 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:33.790774 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:33.790782 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:33.791023 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:33.791803 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:33.791853 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:33.793178 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:33.797264 (+ 4086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:33.797273 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:33.797277 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:33.797915 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:33.798336 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:33.798341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:33.798345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:33.798481 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:33.799108 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:33.799114 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:33.799459 (+ 345us) client_negotiation.cc:769] Sending connection context
0928 02:17:33.799640 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 02:17:33.800796 (+ 1156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":331,"threads_started":1}
I20260928 02:17:33.803130 6609 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:37925
I20260928 02:17:33.803202 6743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:37925 every 8 connection(s)
I20260928 02:17:33.804524 6609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:17:33.807011 6744 heartbeater.cc:347] Connected to a master server at 127.4.140.254:34297
I20260928 02:17:33.807245 6744 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:33.807957 6744 heartbeater.cc:511] Master 127.4.140.254:34297 requested a full tablet report, sending...
I20260928 02:17:33.809427 6398 ts_manager.cc:194] Registered new tserver with Master: afc578d2b43a495382ccde49ffa32ad1 (127.4.140.194:37925)
I20260928 02:17:33.810339 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 6609
I20260928 02:17:33.810532 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:17:33.810550 6398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:48071
I20260928 02:17:33.811101 4659 external_mini_cluster.cc:1428] Setting key 67cc8c98695d9c363ec1a47702661b68
WARNING: no policy specified for kudu/127.4.140.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
I20260928 02:17:33.853466 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:34297
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:44425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:34.186668 6748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:34.186960 6748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:34.187060 6748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:34.194507 6748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:34.194618 6748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:34.194741 6748 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:17:34.194815 6748 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:17:34.194928 6748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:17:34.205116 6748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:44425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.140.254:34297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:34.206485 6748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:34.208314 6748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:34.222025 6753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:34.221989 6756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:34.222077 6754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:34.222980 6748 server_base.cc:1034] running on GCE node
I20260928 02:17:34.223626 6748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:34.224696 6748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:34.225934 6748 hybrid_clock.cc:648] HybridClock initialized: now 1790561854225883 us; error 43 us; skew 500 ppm
I20260928 02:17:34.226259 6748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:34 dist-test-slave-h37r krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561854, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:34.229481 6748 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:17:34.231055 6748 webserver.cc:460] Webserver started at http://127.4.140.195:38199/ using document root <none> and password file <none>
I20260928 02:17:34.231742 6748 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:34.231889 6748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:34.232287 6748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:34.235493 6748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "6ef1f9420dd940e78cee3642e8b8de4e"
format_stamp: "Formatted at 2026-09-28 02:17:34 on dist-test-slave-h37r"
server_key: "d2d697b0e9142e6d5169b9b3549979e5"
server_key_iv: "1500677f724618b686b43121acef1992"
server_key_version: "encryptionkey@0"
I20260928 02:17:34.236349 6748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "6ef1f9420dd940e78cee3642e8b8de4e"
format_stamp: "Formatted at 2026-09-28 02:17:34 on dist-test-slave-h37r"
server_key: "d2d697b0e9142e6d5169b9b3549979e5"
server_key_iv: "1500677f724618b686b43121acef1992"
server_key_version: "encryptionkey@0"
I20260928 02:17:34.243176 6748 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 02:17:34.246738 6763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:34.248172 6748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:17:34.248350 6748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "6ef1f9420dd940e78cee3642e8b8de4e"
format_stamp: "Formatted at 2026-09-28 02:17:34 on dist-test-slave-h37r"
server_key: "d2d697b0e9142e6d5169b9b3549979e5"
server_key_iv: "1500677f724618b686b43121acef1992"
server_key_version: "encryptionkey@0"
I20260928 02:17:34.248522 6748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:34.265419 6748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:34.270097 6748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:34.270399 6748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:34.273442 6748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:34.273566 6748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:34.273708 6748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:34.273799 6748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:17:34 dist-test-slave-h37r krb5kdc[6353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561854, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:34.294402 6605 heartbeater.cc:503] Master 127.4.140.254:34297 was elected leader, sending a full tablet report...
I20260928 02:17:34.300647 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.274312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:36363 (local address 127.4.140.254:34297)
0928 02:17:34.274516 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:34.274522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:34.275863 (+ 1341us) server_negotiation.cc:407] Connection header received
0928 02:17:34.278224 (+ 2361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:34.278232 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:34.278346 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:34.278474 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:34.281949 (+ 3475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.283056 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.286272 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.286867 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.286977 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.291811 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:34.291816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:34.291819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:34.291880 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:34.295260 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:34.296346 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:34.296350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:34.296352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:34.296482 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:34.296974 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:34.296978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:34.296981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:34.297325 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:34.297473 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:34.300109 (+ 2636us) server_negotiation.cc:299] Negotiation successful
0928 02:17:34.300323 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:17:34.300864 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.274773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:34297 (local address 127.4.140.195:36363)
0928 02:17:34.275676 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:34.275724 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:34.277837 (+ 2113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:34.278773 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:34.278787 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:34.279640 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:34.281698 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.281728 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.283270 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.283278 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.286069 (+ 2791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.286087 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.287203 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.287208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.287430 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.288332 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:34.288377 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:34.291570 (+ 3193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:34.295410 (+ 3840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:34.295418 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:34.295433 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:34.296206 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:34.296594 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:34.296599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:34.296603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:34.296838 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:34.297454 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:34.297461 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:34.298010 (+ 549us) client_negotiation.cc:769] Sending connection context
0928 02:17:34.298367 (+ 357us) client_negotiation.cc:240] Negotiation successful
0928 02:17:34.298979 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":396,"threads_started":1}
I20260928 02:17:34.320981 6748 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:38725
I20260928 02:17:34.320994 6882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:38725 every 8 connection(s)
I20260928 02:17:34.322643 6748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:17:34.325148 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.308320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:48311 (local address 127.4.140.254:34297)
0928 02:17:34.308500 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:34.308506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:34.309229 (+ 723us) server_negotiation.cc:407] Connection header received
0928 02:17:34.309460 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:34.309466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:34.309551 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:34.309656 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:34.310999 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.312051 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.314478 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.315422 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.315599 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.317843 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:34.317849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:34.317853 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:34.317891 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:34.321846 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:34.322737 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:34.322743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:34.322745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:34.322925 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:34.323357 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:34.323363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:34.323367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:34.323722 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:34.323858 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:34.324489 (+ 631us) server_negotiation.cc:299] Negotiation successful
0928 02:17:34.324720 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:17:34.325232 6883 heartbeater.cc:347] Connected to a master server at 127.4.140.254:34297
I20260928 02:17:34.325524 6883 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:34.326306 6858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.308445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:34297 (local address 127.4.140.195:48311)
0928 02:17:34.309114 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:34.309130 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:34.309300 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:34.309681 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:34.309685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:34.310367 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:34.310851 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.310871 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.312196 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.312201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.314320 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.314331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.315640 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.315644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.315755 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.316519 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:34.316549 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:34.317659 (+ 1110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:34.321994 (+ 4335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:34.322000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:34.322003 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:34.322600 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:34.323045 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:34.323051 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:34.323055 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:34.323210 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:34.323995 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:34.324000 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:34.324206 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 02:17:34.325406 (+ 1200us) client_negotiation.cc:240] Negotiation successful
0928 02:17:34.325784 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":614,"threads_started":1}
I20260928 02:17:34.328013 6883 heartbeater.cc:511] Master 127.4.140.254:34297 requested a full tablet report, sending...
I20260928 02:17:34.329962 6398 ts_manager.cc:194] Registered new tserver with Master: 6ef1f9420dd940e78cee3642e8b8de4e (127.4.140.195:38725)
I20260928 02:17:34.331483 6398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:36363
I20260928 02:17:34.332909 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 6748
I20260928 02:17:34.333078 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:17:34.333472 4659 external_mini_cluster.cc:1428] Setting key f8fcbd9ac33e04477b4393997eb353cf
I20260928 02:17:34.336478 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:17:34 dist-test-slave-h37r krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:17:34 dist-test-slave-h37r krb5kdc[6353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:34.384652 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.370394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41008 (local address 127.4.140.254:34297)
0928 02:17:34.370641 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:34.370647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:34.370905 (+ 258us) server_negotiation.cc:407] Connection header received
0928 02:17:34.371058 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:34.371062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:34.371125 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:34.371212 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:34.372978 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.373752 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.375175 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.375659 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.375769 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.379840 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:34.379854 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:34.379857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:34.379947 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:34.382413 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:34.383010 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:34.383014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:34.383016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:34.383125 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:34.383427 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:34.383430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:34.383432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:34.383685 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:34.383862 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:34.384140 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 02:17:34.384302 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 02:17:34.388564 6398 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:17:34.395378 6398 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:17:34.418339 6901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.410206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60032 (local address 127.4.140.194:37925)
0928 02:17:34.410903 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:34.410914 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:34.410945 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:17:34.411044 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:34.411053 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:34.411289 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:34.411496 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:34.412425 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.413540 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.416064 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.417079 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.417229 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.417444 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:34.417813 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 02:17:34.418008 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":397,"threads_started":1}
I20260928 02:17:34.418521 6896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.407708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:35461 (local address 127.0.0.1:34412)
0928 02:17:34.408858 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:34.408911 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:34.409119 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:34.410466 (+ 1347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:34.410475 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:34.410517 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:34.410936 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.410951 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.412943 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.412948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.415493 (+ 2545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.415509 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.417567 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.417573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.417782 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.417814 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:17:34.417876 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:17:34.418005 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":7,"thread_start_us":394,"threads_started":1}
I20260928 02:17:34.419015 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.407912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34412 (local address 127.4.140.193:35461)
0928 02:17:34.409731 (+ 1819us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:34.409742 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:34.409779 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:17:34.409900 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:34.409907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:34.410199 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:34.411153 (+ 954us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:34.411371 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.412793 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.415707 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.417319 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.418153 (+ 834us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.418369 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:34.418453 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 02:17:34.418680 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1666,"thread_start_us":1564,"threads_started":1}
I20260928 02:17:34.419072 6900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.409967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:37925 (local address 127.0.0.1:60032)
0928 02:17:34.410472 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:34.410487 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:34.410590 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:34.411719 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:34.411724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:34.411766 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:34.412236 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.412246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.413676 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.413682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.415821 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.415839 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.417302 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.417309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.417536 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.417602 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 02:17:34.417755 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 02:17:34.417961 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":81,"thread_start_us":332,"threads_started":1}
I20260928 02:17:34.420318 6898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.408954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:38725 (local address 127.0.0.1:44570)
0928 02:17:34.409533 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:34.409552 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:34.409663 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:34.411616 (+ 1953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:34.411621 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:34.411645 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:34.412204 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.412222 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.414514 (+ 2292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.414520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.416879 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.416894 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.419427 (+ 2533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.419435 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.419636 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.419673 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:17:34.419747 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:17:34.419875 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":63,"thread_start_us":289,"threads_started":1}
I20260928 02:17:34.420812 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.409272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.4.140.195:38725)
0928 02:17:34.411076 (+ 1804us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:34.411085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:34.411113 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:17:34.411196 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:34.411203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:34.411405 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:34.412439 (+ 1034us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:34.412691 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.414343 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.417061 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.418381 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.420050 (+ 1669us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.420269 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:34.420342 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 02:17:34.420496 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1642,"thread_start_us":838,"threads_started":1}
I20260928 02:17:34.422811 6535 tablet_service.cc:1467] Processing CreateTablet for tablet a332063b1f2c40758377dc2d244baaa3 (DEFAULT_TABLE table=test-table [id=b06d5ad5df18487b85f4859a7feab347]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:34.422827 6674 tablet_service.cc:1467] Processing CreateTablet for tablet a332063b1f2c40758377dc2d244baaa3 (DEFAULT_TABLE table=test-table [id=b06d5ad5df18487b85f4859a7feab347]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:34.424196 6674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a332063b1f2c40758377dc2d244baaa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:34.424206 6535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a332063b1f2c40758377dc2d244baaa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:34.424312 6813 tablet_service.cc:1467] Processing CreateTablet for tablet a332063b1f2c40758377dc2d244baaa3 (DEFAULT_TABLE table=test-table [id=b06d5ad5df18487b85f4859a7feab347]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:34.425578 6813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a332063b1f2c40758377dc2d244baaa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:34.436010 6902 tablet_bootstrap.cc:492] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55: Bootstrap starting.
I20260928 02:17:34.438894 6903 tablet_bootstrap.cc:492] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1: Bootstrap starting.
I20260928 02:17:34.440855 6904 tablet_bootstrap.cc:492] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e: Bootstrap starting.
I20260928 02:17:34.440923 6902 tablet_bootstrap.cc:654] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:34.442643 6902 log.cc:824] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:34.442768 6903 tablet_bootstrap.cc:654] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:34.444079 6904 tablet_bootstrap.cc:654] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:34.444060 6903 log.cc:824] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:34.445225 6904 log.cc:824] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:34.445974 6902 tablet_bootstrap.cc:492] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55: No bootstrap required, opened a new log
I20260928 02:17:34.446333 6902 ts_tablet_manager.cc:1397] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260928 02:17:34.447541 6903 tablet_bootstrap.cc:492] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1: No bootstrap required, opened a new log
I20260928 02:17:34.447826 6903 ts_tablet_manager.cc:1397] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 02:17:34.448235 6904 tablet_bootstrap.cc:492] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e: No bootstrap required, opened a new log
I20260928 02:17:34.448495 6904 ts_tablet_manager.cc:1397] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260928 02:17:34.455789 6904 raft_consensus.cc:348] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } }
I20260928 02:17:34.456233 6904 raft_consensus.cc:374] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:34.456344 6904 raft_consensus.cc:729] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef1f9420dd940e78cee3642e8b8de4e, State: Initialized, Role: FOLLOWER
I20260928 02:17:34.457113 6904 consensus_queue.cc:260] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } }
I20260928 02:17:34.457840 6902 raft_consensus.cc:348] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } }
I20260928 02:17:34.458355 6902 raft_consensus.cc:374] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:34.458506 6902 raft_consensus.cc:729] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554900af951f497489f6a9b68f810d55, State: Initialized, Role: FOLLOWER
I20260928 02:17:34.458729 6883 heartbeater.cc:503] Master 127.4.140.254:34297 was elected leader, sending a full tablet report...
I20260928 02:17:34.458894 6903 raft_consensus.cc:348] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } }
I20260928 02:17:34.459347 6903 raft_consensus.cc:374] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:34.459448 6903 raft_consensus.cc:729] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afc578d2b43a495382ccde49ffa32ad1, State: Initialized, Role: FOLLOWER
I20260928 02:17:34.459334 6902 consensus_queue.cc:260] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } }
I20260928 02:17:34.460151 6903 consensus_queue.cc:260] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } }
I20260928 02:17:34.461426 6744 heartbeater.cc:503] Master 127.4.140.254:34297 was elected leader, sending a full tablet report...
I20260928 02:17:34.462333 6903 ts_tablet_manager.cc:1428] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260928 02:17:34.464159 6904 ts_tablet_manager.cc:1428] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e: Time spent starting tablet: real 0.016s user 0.011s sys 0.000s
I20260928 02:17:34.464286 6902 ts_tablet_manager.cc:1428] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55: Time spent starting tablet: real 0.018s user 0.013s sys 0.003s
W20260928 02:17:34.534551 6606 tablet.cc:2369] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:17:34.556895 6745 tablet.cc:2369] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:17:34.583073 6884 tablet.cc:2369] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:17:34.717133 6910 raft_consensus.cc:482] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:17:34.717437 6910 raft_consensus.cc:504] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } }
I20260928 02:17:34.719175 6910 leader_election.cc:288] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 554900af951f497489f6a9b68f810d55 (127.4.140.193:35461), 6ef1f9420dd940e78cee3642e8b8de4e (127.4.140.195:38725)
I20260928 02:17:34.725617 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.719579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:43455 (local address 127.4.140.193:35461)
0928 02:17:34.719837 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:34.719845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:34.720309 (+ 464us) server_negotiation.cc:407] Connection header received
0928 02:17:34.720554 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:34.720569 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:34.720662 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:34.720758 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:34.721369 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.722070 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.723861 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.724706 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.724895 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.724968 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:34.725245 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 02:17:34.725376 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 02:17:34.725746 6911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.719392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:35461 (local address 127.4.140.194:43455)
0928 02:17:34.720209 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:34.720229 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:34.720395 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:34.720849 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:34.720853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:34.720876 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:34.721215 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.721223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.722199 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.722207 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.723689 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.723701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.724915 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.724921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.725077 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.725111 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:17:34.725219 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 02:17:34.725349 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":374,"threads_started":1}
I20260928 02:17:34.726079 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.720533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:36577 (local address 127.4.140.195:38725)
0928 02:17:34.720735 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:34.720741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:34.721086 (+ 345us) server_negotiation.cc:407] Connection header received
0928 02:17:34.721281 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:34.721285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:34.721336 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:34.721436 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:34.721963 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.722645 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.724408 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.725164 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.725343 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.725427 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:34.725705 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 02:17:34.725826 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:17:34.726163 6912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.720217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:38725 (local address 127.4.140.194:36577)
0928 02:17:34.720975 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:34.720998 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:34.721136 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:34.721448 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:34.721452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:34.721475 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:34.721806 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.721814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.722794 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.722801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.724251 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.724262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.725379 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.725386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.725551 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.725574 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:17:34.725682 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 02:17:34.725808 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":392,"threads_started":1}
I20260928 02:17:34.726781 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a332063b1f2c40758377dc2d244baaa3" candidate_uuid: "afc578d2b43a495382ccde49ffa32ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554900af951f497489f6a9b68f810d55" is_pre_election: true
I20260928 02:17:34.726991 6833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a332063b1f2c40758377dc2d244baaa3" candidate_uuid: "afc578d2b43a495382ccde49ffa32ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" is_pre_election: true
I20260928 02:17:34.727293 6560 raft_consensus.cc:2454] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afc578d2b43a495382ccde49ffa32ad1 in term 0.
I20260928 02:17:34.727536 6833 raft_consensus.cc:2454] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afc578d2b43a495382ccde49ffa32ad1 in term 0.
I20260928 02:17:34.728035 6626 leader_election.cc:302] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 554900af951f497489f6a9b68f810d55, afc578d2b43a495382ccde49ffa32ad1; no voters:
I20260928 02:17:34.728502 6910 raft_consensus.cc:2781] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:17:34.728643 6910 raft_consensus.cc:482] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:17:34.728786 6910 raft_consensus.cc:3037] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:34.731040 6910 raft_consensus.cc:504] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } }
I20260928 02:17:34.731616 6910 leader_election.cc:288] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [CANDIDATE]: Term 1 election: Requested vote from peers 554900af951f497489f6a9b68f810d55 (127.4.140.193:35461), 6ef1f9420dd940e78cee3642e8b8de4e (127.4.140.195:38725)
I20260928 02:17:34.732271 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a332063b1f2c40758377dc2d244baaa3" candidate_uuid: "afc578d2b43a495382ccde49ffa32ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554900af951f497489f6a9b68f810d55"
I20260928 02:17:34.732343 6833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a332063b1f2c40758377dc2d244baaa3" candidate_uuid: "afc578d2b43a495382ccde49ffa32ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef1f9420dd940e78cee3642e8b8de4e"
I20260928 02:17:34.732533 6560 raft_consensus.cc:3037] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:34.732600 6833 raft_consensus.cc:3037] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:34.734704 6560 raft_consensus.cc:2454] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afc578d2b43a495382ccde49ffa32ad1 in term 1.
I20260928 02:17:34.734717 6833 raft_consensus.cc:2454] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afc578d2b43a495382ccde49ffa32ad1 in term 1.
I20260928 02:17:34.735113 6626 leader_election.cc:302] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 554900af951f497489f6a9b68f810d55, afc578d2b43a495382ccde49ffa32ad1; no voters:
I20260928 02:17:34.735397 6910 raft_consensus.cc:2781] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:34.735853 6910 raft_consensus.cc:686] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [term 1 LEADER]: Becoming Leader. State: Replica: afc578d2b43a495382ccde49ffa32ad1, State: Running, Role: LEADER
I20260928 02:17:34.736398 6910 consensus_queue.cc:237] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } }
I20260928 02:17:34.741081 6397 catalog_manager.cc:5495] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 reported cstate change: term changed from 0 to 1, leader changed from <none> to afc578d2b43a495382ccde49ffa32ad1 (127.4.140.194). New cstate: current_term: 1 leader_uuid: "afc578d2b43a495382ccde49ffa32ad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } health_report { overall_health: HEALTHY } } }
Sep 28 02:17:34 dist-test-slave-h37r krb5kdc[6353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561852, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:34.820015 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.806654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41022 (local address 127.4.140.254:34297)
0928 02:17:34.806830 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:34.806835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:34.807371 (+ 536us) server_negotiation.cc:407] Connection header received
0928 02:17:34.807579 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:34.807589 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:34.807642 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:34.807744 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:34.809067 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.809801 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.811075 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.811483 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.811625 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.814959 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:34.814964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:34.814967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:34.815005 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:34.817336 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:34.818005 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:34.818013 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:34.818017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:34.818131 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:34.818488 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:34.818495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:34.818506 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:34.818780 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:34.818929 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:34.819362 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 02:17:34.819660 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":60}
I20260928 02:17:34.820048 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.806680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:34297 (local address 127.0.0.1:41022)
0928 02:17:34.807255 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:34.807271 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:34.807446 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:34.807790 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:34.807794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:34.808636 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:34.808904 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.808913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.809938 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.809946 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.810937 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.810948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.811669 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.811674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.811828 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.812551 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:34.812569 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:34.814773 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:34.817459 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:34.817472 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:34.817480 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:34.817892 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:34.818235 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:34.818241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:34.818244 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:34.818385 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:34.818921 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:34.818929 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:34.819080 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 02:17:34.819232 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 02:17:34.819523 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":337,"threads_started":1}
I20260928 02:17:34.825045 6397 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:41022:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:17:34.826064 6397 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:17:34.832571 6535 tablet_service.cc:1467] Processing CreateTablet for tablet a914bc78bf064d098b3da6517aaeb6c7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c122f7bd4b3c49b3b9a5934ed9f530eb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:17:34.832792 6813 tablet_service.cc:1467] Processing CreateTablet for tablet a914bc78bf064d098b3da6517aaeb6c7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c122f7bd4b3c49b3b9a5934ed9f530eb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:17:34.832652 6674 tablet_service.cc:1467] Processing CreateTablet for tablet a914bc78bf064d098b3da6517aaeb6c7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c122f7bd4b3c49b3b9a5934ed9f530eb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:17:34.833493 6535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a914bc78bf064d098b3da6517aaeb6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:34.833516 6813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a914bc78bf064d098b3da6517aaeb6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:34.833531 6674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a914bc78bf064d098b3da6517aaeb6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:34.838724 6902 tablet_bootstrap.cc:492] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55: Bootstrap starting.
I20260928 02:17:34.840364 6902 tablet_bootstrap.cc:654] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:34.840602 6904 tablet_bootstrap.cc:492] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e: Bootstrap starting.
I20260928 02:17:34.840605 6903 tablet_bootstrap.cc:492] T a914bc78bf064d098b3da6517aaeb6c7 P afc578d2b43a495382ccde49ffa32ad1: Bootstrap starting.
I20260928 02:17:34.842043 6904 tablet_bootstrap.cc:654] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:34.842653 6903 tablet_bootstrap.cc:654] T a914bc78bf064d098b3da6517aaeb6c7 P afc578d2b43a495382ccde49ffa32ad1: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:34.844581 6904 tablet_bootstrap.cc:492] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e: No bootstrap required, opened a new log
I20260928 02:17:34.844797 6904 ts_tablet_manager.cc:1397] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260928 02:17:34.845463 6902 tablet_bootstrap.cc:492] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55: No bootstrap required, opened a new log
I20260928 02:17:34.845611 6902 ts_tablet_manager.cc:1397] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260928 02:17:34.845750 6904 raft_consensus.cc:348] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:34.845994 6904 raft_consensus.cc:374] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:34.846061 6903 tablet_bootstrap.cc:492] T a914bc78bf064d098b3da6517aaeb6c7 P afc578d2b43a495382ccde49ffa32ad1: No bootstrap required, opened a new log
I20260928 02:17:34.846113 6904 raft_consensus.cc:729] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef1f9420dd940e78cee3642e8b8de4e, State: Initialized, Role: FOLLOWER
I20260928 02:17:34.846241 6903 ts_tablet_manager.cc:1397] T a914bc78bf064d098b3da6517aaeb6c7 P afc578d2b43a495382ccde49ffa32ad1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260928 02:17:34.846356 6904 consensus_queue.cc:260] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:34.846382 6902 raft_consensus.cc:348] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:34.846647 6902 raft_consensus.cc:374] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:34.846740 6902 raft_consensus.cc:729] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554900af951f497489f6a9b68f810d55, State: Initialized, Role: FOLLOWER
I20260928 02:17:34.846820 6908 tablet_replica.cc:439] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } } }
I20260928 02:17:34.846980 6902 consensus_queue.cc:260] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:34.847133 6908 tablet_replica.cc:442] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:17:34.847055 6903 raft_consensus.cc:348] T a914bc78bf064d098b3da6517aaeb6c7 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:34.847488 6909 tablet_replica.cc:439] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } } }
I20260928 02:17:34.847863 6909 tablet_replica.cc:442] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:17:34.848106 6903 raft_consensus.cc:374] T a914bc78bf064d098b3da6517aaeb6c7 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:34.848212 6904 ts_tablet_manager.cc:1428] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:17:34.848256 6903 raft_consensus.cc:729] T a914bc78bf064d098b3da6517aaeb6c7 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afc578d2b43a495382ccde49ffa32ad1, State: Initialized, Role: FOLLOWER
I20260928 02:17:34.848515 6903 consensus_queue.cc:260] T a914bc78bf064d098b3da6517aaeb6c7 P afc578d2b43a495382ccde49ffa32ad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:34.848954 6910 tablet_replica.cc:439] T a914bc78bf064d098b3da6517aaeb6c7 P afc578d2b43a495382ccde49ffa32ad1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } } }
I20260928 02:17:34.849231 6910 tablet_replica.cc:442] T a914bc78bf064d098b3da6517aaeb6c7 P afc578d2b43a495382ccde49ffa32ad1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:17:34.849773 6903 ts_tablet_manager.cc:1428] T a914bc78bf064d098b3da6517aaeb6c7 P afc578d2b43a495382ccde49ffa32ad1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:17:34.849915 6902 ts_tablet_manager.cc:1428] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260928 02:17:34.902706 6909 raft_consensus.cc:482] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:17:34.903021 6909 raft_consensus.cc:504] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:34.905055 6909 leader_election.cc:288] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afc578d2b43a495382ccde49ffa32ad1 (127.4.140.194:37925), 6ef1f9420dd940e78cee3642e8b8de4e (127.4.140.195:38725)
I20260928 02:17:34.909065 6908 raft_consensus.cc:482] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:17:34.909263 6908 raft_consensus.cc:504] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:34.911069 6908 leader_election.cc:288] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 554900af951f497489f6a9b68f810d55 (127.4.140.193:35461), afc578d2b43a495382ccde49ffa32ad1 (127.4.140.194:37925)
I20260928 02:17:34.913157 6901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.905337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:33997 (local address 127.4.140.194:37925)
0928 02:17:34.905584 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:34.905591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:34.906328 (+ 737us) server_negotiation.cc:407] Connection header received
0928 02:17:34.906513 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:34.906517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:34.906575 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:34.906677 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:34.907407 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.908227 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.910490 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.911809 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.912026 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.912123 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:34.912594 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 02:17:34.912782 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260928 02:17:34.914408 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.905603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:45515 (local address 127.4.140.195:38725)
0928 02:17:34.905809 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:34.905817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:34.906215 (+ 398us) server_negotiation.cc:407] Connection header received
0928 02:17:34.906471 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:34.906479 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:34.906559 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:34.906634 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:34.907337 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.908235 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.910725 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.912140 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.912749 (+ 609us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.912828 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:34.913960 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0928 02:17:34.914111 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:17:34.915175 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.905329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:38725 (local address 127.4.140.193:45515)
0928 02:17:34.906112 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:34.906126 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:34.906287 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:34.906681 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:34.906686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:34.906708 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:34.907147 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.907160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.908403 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.908412 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.910550 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.910567 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.913585 (+ 3018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.913589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.913771 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.913793 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:17:34.914571 (+ 778us) client_negotiation.cc:240] Negotiation successful
0928 02:17:34.914730 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":95,"thread_start_us":443,"threads_started":1}
I20260928 02:17:34.915838 6925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.905397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:37925 (local address 127.4.140.193:33997)
0928 02:17:34.906211 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:34.906238 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:34.906380 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:34.906720 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:34.906725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:34.906750 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:34.907208 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.907218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.908365 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.908373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.910307 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.910319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.912055 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.912064 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.912385 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.912432 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 02:17:34.915317 (+ 2885us) client_negotiation.cc:240] Negotiation successful
0928 02:17:34.915476 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":74,"thread_start_us":462,"threads_started":1}
I20260928 02:17:34.916460 6833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a914bc78bf064d098b3da6517aaeb6c7" candidate_uuid: "554900af951f497489f6a9b68f810d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" is_pre_election: true
I20260928 02:17:34.916751 6833 raft_consensus.cc:2454] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 554900af951f497489f6a9b68f810d55 in term 0.
I20260928 02:17:34.917199 6486 leader_election.cc:302] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 554900af951f497489f6a9b68f810d55, 6ef1f9420dd940e78cee3642e8b8de4e; no voters:
I20260928 02:17:34.916982 6698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a914bc78bf064d098b3da6517aaeb6c7" candidate_uuid: "554900af951f497489f6a9b68f810d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afc578d2b43a495382ccde49ffa32ad1" is_pre_election: true
I20260928 02:17:34.917579 6909 raft_consensus.cc:2781] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:17:34.917546 6698 raft_consensus.cc:2454] T a914bc78bf064d098b3da6517aaeb6c7 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 554900af951f497489f6a9b68f810d55 in term 0.
I20260928 02:17:34.917729 6909 raft_consensus.cc:482] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:17:34.917841 6909 raft_consensus.cc:3037] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:34.921383 6909 raft_consensus.cc:504] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:34.922439 6901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.912329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:57967 (local address 127.4.140.194:37925)
0928 02:17:34.913443 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:34.913448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:34.913670 (+ 222us) server_negotiation.cc:407] Connection header received
0928 02:17:34.913883 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:34.913889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:34.913980 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:34.914060 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:34.914942 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.916536 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.920550 (+ 4014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.921430 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.921526 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.921572 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:34.922027 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 02:17:34.922143 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018,"threads_started":1}
I20260928 02:17:34.922518 6927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.912287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:37925 (local address 127.4.140.195:57967)
0928 02:17:34.913539 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:34.913563 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:34.913723 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:34.914195 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:34.914199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:34.914226 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:34.914754 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.914767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.916730 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.916738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.920244 (+ 3506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.920259 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.921706 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.921714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.921878 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.921901 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:17:34.922003 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:17:34.922126 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1095,"mutex_wait_us":1181,"thread_start_us":1084,"threads_started":1}
I20260928 02:17:34.922676 6698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a914bc78bf064d098b3da6517aaeb6c7" candidate_uuid: "554900af951f497489f6a9b68f810d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afc578d2b43a495382ccde49ffa32ad1"
I20260928 02:17:34.922896 6698 raft_consensus.cc:3037] T a914bc78bf064d098b3da6517aaeb6c7 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:34.923106 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.912559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:34929 (local address 127.4.140.193:35461)
0928 02:17:34.912724 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:34.912741 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:34.912760 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:17:34.912817 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:34.912822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:34.912913 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:34.913032 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:34.914575 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.916161 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.920264 (+ 4103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.921746 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.921929 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.922003 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:34.922679 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 02:17:34.922826 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:17:34.923877 6926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.911151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:35461 (local address 127.4.140.195:34929)
0928 02:17:34.911701 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:34.911716 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:34.911802 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:34.913915 (+ 2113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:34.913924 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:34.913958 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:34.914343 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.914360 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.916337 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.916342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.920064 (+ 3722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.920081 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.922222 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.922235 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.922461 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.922513 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 02:17:34.923333 (+ 820us) client_negotiation.cc:240] Negotiation successful
0928 02:17:34.923472 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":3,"thread_start_us":243,"threads_started":1}
I20260928 02:17:34.924199 6909 leader_election.cc:288] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 [CANDIDATE]: Term 1 election: Requested vote from peers afc578d2b43a495382ccde49ffa32ad1 (127.4.140.194:37925), 6ef1f9420dd940e78cee3642e8b8de4e (127.4.140.195:38725)
I20260928 02:17:34.924898 6833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a914bc78bf064d098b3da6517aaeb6c7" candidate_uuid: "554900af951f497489f6a9b68f810d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef1f9420dd940e78cee3642e8b8de4e"
I20260928 02:17:34.925170 6833 raft_consensus.cc:3037] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:34.925879 6698 raft_consensus.cc:2454] T a914bc78bf064d098b3da6517aaeb6c7 P afc578d2b43a495382ccde49ffa32ad1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 554900af951f497489f6a9b68f810d55 in term 1.
I20260928 02:17:34.926319 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a914bc78bf064d098b3da6517aaeb6c7" candidate_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554900af951f497489f6a9b68f810d55" is_pre_election: true
I20260928 02:17:34.926388 6487 leader_election.cc:302] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 554900af951f497489f6a9b68f810d55, afc578d2b43a495382ccde49ffa32ad1; no voters:
I20260928 02:17:34.926677 6560 raft_consensus.cc:2379] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ef1f9420dd940e78cee3642e8b8de4e in current term 1: Already voted for candidate 554900af951f497489f6a9b68f810d55 in this term.
I20260928 02:17:34.926987 6909 raft_consensus.cc:2781] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:34.928072 6909 raft_consensus.cc:686] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 [term 1 LEADER]: Becoming Leader. State: Replica: 554900af951f497489f6a9b68f810d55, State: Running, Role: LEADER
I20260928 02:17:34.928675 6909 consensus_queue.cc:237] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:34.929078 6833 raft_consensus.cc:2454] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 554900af951f497489f6a9b68f810d55 in term 1.
I20260928 02:17:34.929414 6698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a914bc78bf064d098b3da6517aaeb6c7" candidate_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afc578d2b43a495382ccde49ffa32ad1" is_pre_election: true
I20260928 02:17:34.929731 6698 raft_consensus.cc:2379] T a914bc78bf064d098b3da6517aaeb6c7 P afc578d2b43a495382ccde49ffa32ad1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ef1f9420dd940e78cee3642e8b8de4e in current term 1: Already voted for candidate 554900af951f497489f6a9b68f810d55 in this term.
I20260928 02:17:34.930347 6766 leader_election.cc:302] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ef1f9420dd940e78cee3642e8b8de4e; no voters: 554900af951f497489f6a9b68f810d55, afc578d2b43a495382ccde49ffa32ad1
I20260928 02:17:34.930711 6908 raft_consensus.cc:2726] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:17:34.932336 6929 tablet_replica.cc:439] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55: TxnStatusTablet state changed. Reason: New leader 554900af951f497489f6a9b68f810d55. Latest consensus state: current_term: 1 leader_uuid: "554900af951f497489f6a9b68f810d55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } } }
I20260928 02:17:34.932682 6929 tablet_replica.cc:442] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:17:34.933552 6931 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:17:34.933833 6931 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:17:34.934340 6398 catalog_manager.cc:5495] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 554900af951f497489f6a9b68f810d55 (127.4.140.193). New cstate: current_term: 1 leader_uuid: "554900af951f497489f6a9b68f810d55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } health_report { overall_health: UNKNOWN } } }
I20260928 02:17:34.953585 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.947314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:35461 (local address 127.0.0.1:34422)
0928 02:17:34.947553 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:34.947570 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:34.947667 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:34.948197 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:34.948201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:34.948213 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:17:34.948357 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.948387 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.949801 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.949805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.950854 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:34.950867 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.951855 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.951859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:34.952091 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.952149 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:17:34.953103 (+ 954us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:17:34.953137 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:17:34.953202 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:17:34.953302 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 02:17:34.953688 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.947418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34422 (local address 127.4.140.193:35461)
0928 02:17:34.947590 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:34.947596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:34.947621 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:17:34.947871 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:34.947877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:34.948026 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:34.948185 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:17:34.948510 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.949655 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.951000 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.951665 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:34.951855 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:34.952329 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:17:34.952967 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:17:34.953117 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:34.953253 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 02:17:34.953353 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:17:34.960534 6698 raft_consensus.cc:1261] T a914bc78bf064d098b3da6517aaeb6c7 P afc578d2b43a495382ccde49ffa32ad1 [term 1 FOLLOWER]: Refusing update from remote peer 554900af951f497489f6a9b68f810d55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:17:34.960882 6833 raft_consensus.cc:1261] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 FOLLOWER]: Refusing update from remote peer 554900af951f497489f6a9b68f810d55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:17:34.961454 6929 consensus_queue.cc:1035] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:17:34.961829 6909 consensus_queue.cc:1035] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:17:34.972025 6910 tablet_replica.cc:439] T a914bc78bf064d098b3da6517aaeb6c7 P afc578d2b43a495382ccde49ffa32ad1: TxnStatusTablet state changed. Reason: New leader 554900af951f497489f6a9b68f810d55. Latest consensus state: current_term: 1 leader_uuid: "554900af951f497489f6a9b68f810d55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } } }
I20260928 02:17:34.972456 6910 tablet_replica.cc:442] T a914bc78bf064d098b3da6517aaeb6c7 P afc578d2b43a495382ccde49ffa32ad1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:17:34.974422 6908 tablet_replica.cc:439] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e: TxnStatusTablet state changed. Reason: New leader 554900af951f497489f6a9b68f810d55. Latest consensus state: current_term: 1 leader_uuid: "554900af951f497489f6a9b68f810d55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } } }
I20260928 02:17:34.974706 6908 tablet_replica.cc:442] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:17:34.977080 6936 mvcc.cc:205] Tried to move back new op lower bound from 7334141357909729280 to 7334141357793222656. Current Snapshot: MvccSnapshot[applied={T|T < 7334141357909729280}]
I20260928 02:17:34.978715 6909 tablet_replica.cc:439] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "554900af951f497489f6a9b68f810d55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } } }
I20260928 02:17:34.979024 6909 tablet_replica.cc:442] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:17:34.979301 6909 tablet_replica.cc:439] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "554900af951f497489f6a9b68f810d55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } } }
I20260928 02:17:34.979502 6909 tablet_replica.cc:442] T a914bc78bf064d098b3da6517aaeb6c7 P 554900af951f497489f6a9b68f810d55: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:17:34.990243 6908 tablet_replica.cc:439] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "554900af951f497489f6a9b68f810d55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } } }
I20260928 02:17:34.990540 6908 tablet_replica.cc:442] T a914bc78bf064d098b3da6517aaeb6c7 P 6ef1f9420dd940e78cee3642e8b8de4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:17:34.992972 6910 tablet_replica.cc:439] T a914bc78bf064d098b3da6517aaeb6c7 P afc578d2b43a495382ccde49ffa32ad1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "554900af951f497489f6a9b68f810d55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } } }
I20260928 02:17:34.993254 6910 tablet_replica.cc:442] T a914bc78bf064d098b3da6517aaeb6c7 P afc578d2b43a495382ccde49ffa32ad1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:17:35.003041 6928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:34.995967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60038 (local address 127.4.140.194:37925)
0928 02:17:34.996191 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:34.996200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:34.996773 (+ 573us) server_negotiation.cc:407] Connection header received
0928 02:17:34.997007 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:34.997011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:34.997087 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:34.997220 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:17:34.997742 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:34.998481 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:35.000584 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:35.001055 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:35.001157 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:35.001717 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:17:35.002299 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:17:35.002416 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:35.002553 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 02:17:35.002661 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:17:35.021307 6946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:35.015048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:35461 (local address 127.4.140.194:44681)
0928 02:17:35.015601 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:35.015619 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:35.015762 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:35.016140 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:35.016144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:35.016153 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:17:35.016460 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:35.016468 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:35.017918 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:35.017922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:35.019174 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:35.019185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:35.020251 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:35.020256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:35.020372 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:35.020435 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:17:35.020862 (+ 427us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:17:35.020872 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:17:35.020931 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:17:35.021024 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":400,"threads_started":1}
I20260928 02:17:35.021373 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:35.015171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:44681 (local address 127.4.140.193:35461)
0928 02:17:35.015396 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:35.015402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:35.015701 (+ 299us) server_negotiation.cc:407] Connection header received
0928 02:17:35.015930 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:35.015936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:35.016023 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:35.016123 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:17:35.016601 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:35.017778 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:35.019305 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:35.020087 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:35.020221 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:35.020632 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:17:35.020766 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:17:35.020858 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:35.020981 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 02:17:35.021069 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:17:35.031590 6833 raft_consensus.cc:1261] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 FOLLOWER]: Refusing update from remote peer afc578d2b43a495382ccde49ffa32ad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:17:35.031947 6560 raft_consensus.cc:1261] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55 [term 1 FOLLOWER]: Refusing update from remote peer afc578d2b43a495382ccde49ffa32ad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:17:35.032321 6913 consensus_queue.cc:1035] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:17:35.032693 6910 consensus_queue.cc:1035] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:17:35.038906 6935 mvcc.cc:205] Tried to move back new op lower bound from 7334141358203150336 to 7334141357004517376. Current Snapshot: MvccSnapshot[applied={T|T < 7334141358203150336}]
I20260928 02:17:35.044320 6934 mvcc.cc:205] Tried to move back new op lower bound from 7334141358203150336 to 7334141357004517376. Current Snapshot: MvccSnapshot[applied={T|T < 7334141358203150336}]
I20260928 02:17:35.073572 6928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:35.066315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:56177 (local address 127.4.140.194:37925)
0928 02:17:35.066497 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:35.066503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:35.067293 (+ 790us) server_negotiation.cc:407] Connection header received
0928 02:17:35.067513 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:35.067519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:35.067595 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:35.067691 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:17:35.068429 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:35.069544 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:35.071179 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:35.071843 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:35.071988 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:35.072599 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:17:35.072807 (+ 208us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:17:35.072915 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:35.073126 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 02:17:35.073210 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:17:35.073742 6952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:35.066410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:37925 (local address 127.4.140.193:56177)
0928 02:17:35.067199 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:35.067220 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:35.067359 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:35.067720 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:35.067725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:35.067739 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:17:35.068215 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:35.068229 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:35.069698 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:35.069702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:35.071041 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:35.071053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:35.072143 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:35.072150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:35.072316 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:35.072402 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:17:35.072932 (+ 530us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:17:35.072944 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:17:35.073078 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 02:17:35.073208 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":393,"threads_started":1}
I20260928 02:17:35.114725 6396 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:17:35.115070 6396 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:17:35.119253 6396 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e: Sending DeleteTablet for 3 replicas of tablet a332063b1f2c40758377dc2d244baaa3
I20260928 02:17:35.120575 6674 tablet_service.cc:1514] Processing DeleteTablet for tablet a332063b1f2c40758377dc2d244baaa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:35 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:60032
I20260928 02:17:35.120836 6535 tablet_service.cc:1514] Processing DeleteTablet for tablet a332063b1f2c40758377dc2d244baaa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:35 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:34412
I20260928 02:17:35.121315 6813 tablet_service.cc:1514] Processing DeleteTablet for tablet a332063b1f2c40758377dc2d244baaa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:35 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:44570
I20260928 02:17:35.129477 6396 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:17:35.130669 6396 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:17:35.137080 6956 tablet_replica.cc:330] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55: stopping tablet replica
I20260928 02:17:35.137770 6956 raft_consensus.cc:2229] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:17:35.138432 6956 raft_consensus.cc:2258] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:35.143222 6535 tablet_service.cc:1467] Processing CreateTablet for tablet fd7969d6a483430f85133a9ae1d193f5 (DEFAULT_TABLE table=test-table [id=439aed791aa541c28bb47ec1fa816abe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:35.143832 6812 tablet_service.cc:1467] Processing CreateTablet for tablet fd7969d6a483430f85133a9ae1d193f5 (DEFAULT_TABLE table=test-table [id=439aed791aa541c28bb47ec1fa816abe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:35.144330 6535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd7969d6a483430f85133a9ae1d193f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:35.144467 6812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd7969d6a483430f85133a9ae1d193f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:35.148442 6673 tablet_service.cc:1467] Processing CreateTablet for tablet fd7969d6a483430f85133a9ae1d193f5 (DEFAULT_TABLE table=test-table [id=439aed791aa541c28bb47ec1fa816abe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:35.149096 6673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd7969d6a483430f85133a9ae1d193f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:35.151358 6902 tablet_bootstrap.cc:492] T fd7969d6a483430f85133a9ae1d193f5 P 554900af951f497489f6a9b68f810d55: Bootstrap starting.
I20260928 02:17:35.153468 6956 ts_tablet_manager.cc:1905] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:35.154218 6902 tablet_bootstrap.cc:654] T fd7969d6a483430f85133a9ae1d193f5 P 554900af951f497489f6a9b68f810d55: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:35.154268 6957 tablet_replica.cc:330] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e: stopping tablet replica
I20260928 02:17:35.154755 6957 raft_consensus.cc:2229] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:17:35.155169 6957 raft_consensus.cc:2258] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:35.155925 6955 tablet_replica.cc:330] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1: stopping tablet replica
I20260928 02:17:35.156363 6955 raft_consensus.cc:2229] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:17:35.157013 6955 raft_consensus.cc:2258] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:35.158658 6957 ts_tablet_manager.cc:1905] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:35.158665 6955 ts_tablet_manager.cc:1905] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:35.161003 6903 tablet_bootstrap.cc:492] T fd7969d6a483430f85133a9ae1d193f5 P afc578d2b43a495382ccde49ffa32ad1: Bootstrap starting.
I20260928 02:17:35.162724 6903 tablet_bootstrap.cc:654] T fd7969d6a483430f85133a9ae1d193f5 P afc578d2b43a495382ccde49ffa32ad1: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:35.165946 6957 ts_tablet_manager.cc:1918] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:17:35.166556 6904 tablet_bootstrap.cc:492] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e: Bootstrap starting.
I20260928 02:17:35.168757 6904 tablet_bootstrap.cc:654] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:35.172529 6903 tablet_bootstrap.cc:492] T fd7969d6a483430f85133a9ae1d193f5 P afc578d2b43a495382ccde49ffa32ad1: No bootstrap required, opened a new log
I20260928 02:17:35.172783 6903 ts_tablet_manager.cc:1397] T fd7969d6a483430f85133a9ae1d193f5 P afc578d2b43a495382ccde49ffa32ad1: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.001s
I20260928 02:17:35.173741 6903 raft_consensus.cc:348] T fd7969d6a483430f85133a9ae1d193f5 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:35.174206 6903 raft_consensus.cc:374] T fd7969d6a483430f85133a9ae1d193f5 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:35.174345 6903 raft_consensus.cc:729] T fd7969d6a483430f85133a9ae1d193f5 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afc578d2b43a495382ccde49ffa32ad1, State: Initialized, Role: FOLLOWER
I20260928 02:17:35.174774 6956 ts_tablet_manager.cc:1918] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:17:35.174711 6903 consensus_queue.cc:260] T fd7969d6a483430f85133a9ae1d193f5 P afc578d2b43a495382ccde49ffa32ad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:35.175032 6956 log.cc:1196] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/a332063b1f2c40758377dc2d244baaa3
I20260928 02:17:35.175722 6956 ts_tablet_manager.cc:1939] T a332063b1f2c40758377dc2d244baaa3 P 554900af951f497489f6a9b68f810d55: Deleting consensus metadata
I20260928 02:17:35.178735 6903 ts_tablet_manager.cc:1428] T fd7969d6a483430f85133a9ae1d193f5 P afc578d2b43a495382ccde49ffa32ad1: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260928 02:17:35.182698 6957 log.cc:1196] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/a332063b1f2c40758377dc2d244baaa3
I20260928 02:17:35.183557 6957 ts_tablet_manager.cc:1939] T a332063b1f2c40758377dc2d244baaa3 P 6ef1f9420dd940e78cee3642e8b8de4e: Deleting consensus metadata
I20260928 02:17:35.185957 6955 ts_tablet_manager.cc:1918] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:17:35.186811 6902 tablet_bootstrap.cc:492] T fd7969d6a483430f85133a9ae1d193f5 P 554900af951f497489f6a9b68f810d55: No bootstrap required, opened a new log
I20260928 02:17:35.187791 6383 catalog_manager.cc:4841] TS 554900af951f497489f6a9b68f810d55 (127.4.140.193:35461): tablet a332063b1f2c40758377dc2d244baaa3 (table test-table [id=b06d5ad5df18487b85f4859a7feab347]) successfully deleted
I20260928 02:17:35.188264 6902 ts_tablet_manager.cc:1397] T fd7969d6a483430f85133a9ae1d193f5 P 554900af951f497489f6a9b68f810d55: Time spent bootstrapping tablet: real 0.037s user 0.014s sys 0.011s
I20260928 02:17:35.188832 6383 catalog_manager.cc:4841] TS 6ef1f9420dd940e78cee3642e8b8de4e (127.4.140.195:38725): tablet a332063b1f2c40758377dc2d244baaa3 (table test-table [id=b06d5ad5df18487b85f4859a7feab347]) successfully deleted
I20260928 02:17:35.189307 6902 raft_consensus.cc:348] T fd7969d6a483430f85133a9ae1d193f5 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:35.189692 6955 log.cc:1196] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/a332063b1f2c40758377dc2d244baaa3
I20260928 02:17:35.189867 6902 raft_consensus.cc:374] T fd7969d6a483430f85133a9ae1d193f5 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:35.190490 6955 ts_tablet_manager.cc:1939] T a332063b1f2c40758377dc2d244baaa3 P afc578d2b43a495382ccde49ffa32ad1: Deleting consensus metadata
I20260928 02:17:35.190629 6902 raft_consensus.cc:729] T fd7969d6a483430f85133a9ae1d193f5 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554900af951f497489f6a9b68f810d55, State: Initialized, Role: FOLLOWER
I20260928 02:17:35.190973 6902 consensus_queue.cc:260] T fd7969d6a483430f85133a9ae1d193f5 P 554900af951f497489f6a9b68f810d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:35.192579 6904 tablet_bootstrap.cc:492] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e: No bootstrap required, opened a new log
I20260928 02:17:35.192797 6904 ts_tablet_manager.cc:1397] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.003s
I20260928 02:17:35.193914 6904 raft_consensus.cc:348] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:35.194298 6904 raft_consensus.cc:374] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:35.194439 6904 raft_consensus.cc:729] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef1f9420dd940e78cee3642e8b8de4e, State: Initialized, Role: FOLLOWER
I20260928 02:17:35.194789 6902 ts_tablet_manager.cc:1428] T fd7969d6a483430f85133a9ae1d193f5 P 554900af951f497489f6a9b68f810d55: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260928 02:17:35.194830 6904 consensus_queue.cc:260] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:35.198348 6384 catalog_manager.cc:4841] TS afc578d2b43a495382ccde49ffa32ad1 (127.4.140.194:37925): tablet a332063b1f2c40758377dc2d244baaa3 (table test-table [id=b06d5ad5df18487b85f4859a7feab347]) successfully deleted
I20260928 02:17:35.199047 6904 ts_tablet_manager.cc:1428] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260928 02:17:35.241115 6908 raft_consensus.cc:482] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:17:35.241613 6908 raft_consensus.cc:504] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:35.242411 6908 leader_election.cc:288] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 554900af951f497489f6a9b68f810d55 (127.4.140.193:35461), afc578d2b43a495382ccde49ffa32ad1 (127.4.140.194:37925)
I20260928 02:17:35.243172 6698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd7969d6a483430f85133a9ae1d193f5" candidate_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afc578d2b43a495382ccde49ffa32ad1" is_pre_election: true
I20260928 02:17:35.243445 6698 raft_consensus.cc:2454] T fd7969d6a483430f85133a9ae1d193f5 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ef1f9420dd940e78cee3642e8b8de4e in term 0.
I20260928 02:17:35.244108 6766 leader_election.cc:302] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ef1f9420dd940e78cee3642e8b8de4e, afc578d2b43a495382ccde49ffa32ad1; no voters:
I20260928 02:17:35.244745 6908 raft_consensus.cc:2781] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:17:35.244874 6908 raft_consensus.cc:482] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:17:35.244952 6908 raft_consensus.cc:3037] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:35.248148 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd7969d6a483430f85133a9ae1d193f5" candidate_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554900af951f497489f6a9b68f810d55" is_pre_election: true
I20260928 02:17:35.248420 6560 raft_consensus.cc:2454] T fd7969d6a483430f85133a9ae1d193f5 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ef1f9420dd940e78cee3642e8b8de4e in term 0.
I20260928 02:17:35.248941 6908 raft_consensus.cc:504] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:35.249715 6908 leader_election.cc:288] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [CANDIDATE]: Term 1 election: Requested vote from peers 554900af951f497489f6a9b68f810d55 (127.4.140.193:35461), afc578d2b43a495382ccde49ffa32ad1 (127.4.140.194:37925)
I20260928 02:17:35.250104 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd7969d6a483430f85133a9ae1d193f5" candidate_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554900af951f497489f6a9b68f810d55"
I20260928 02:17:35.250313 6560 raft_consensus.cc:3037] T fd7969d6a483430f85133a9ae1d193f5 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:35.250636 6698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd7969d6a483430f85133a9ae1d193f5" candidate_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afc578d2b43a495382ccde49ffa32ad1"
I20260928 02:17:35.250914 6698 raft_consensus.cc:3037] T fd7969d6a483430f85133a9ae1d193f5 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:35.253468 6560 raft_consensus.cc:2454] T fd7969d6a483430f85133a9ae1d193f5 P 554900af951f497489f6a9b68f810d55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef1f9420dd940e78cee3642e8b8de4e in term 1.
I20260928 02:17:35.253988 6765 leader_election.cc:302] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 554900af951f497489f6a9b68f810d55, 6ef1f9420dd940e78cee3642e8b8de4e; no voters:
I20260928 02:17:35.254215 6698 raft_consensus.cc:2454] T fd7969d6a483430f85133a9ae1d193f5 P afc578d2b43a495382ccde49ffa32ad1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef1f9420dd940e78cee3642e8b8de4e in term 1.
I20260928 02:17:35.254426 6908 raft_consensus.cc:2781] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:35.255275 6908 raft_consensus.cc:686] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 LEADER]: Becoming Leader. State: Replica: 6ef1f9420dd940e78cee3642e8b8de4e, State: Running, Role: LEADER
I20260928 02:17:35.255772 6908 consensus_queue.cc:237] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:35.259840 6398 catalog_manager.cc:5495] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ef1f9420dd940e78cee3642e8b8de4e (127.4.140.195). New cstate: current_term: 1 leader_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } health_report { overall_health: HEALTHY } } }
I20260928 02:17:35.302621 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:35.294817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44576 (local address 127.4.140.195:38725)
0928 02:17:35.294999 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:35.295006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:35.295025 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:17:35.295184 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:35.295190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:35.295337 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:35.295492 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:17:35.296107 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:35.297252 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:35.299254 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:35.299969 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:35.300085 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:35.300670 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:17:35.301411 (+ 741us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:17:35.301558 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:35.302166 (+ 608us) server_negotiation.cc:299] Negotiation successful
0928 02:17:35.302265 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:17:35.347342 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:35.333589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:59633 (local address 127.4.140.193:35461)
0928 02:17:35.333823 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:35.333831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:35.340691 (+ 6860us) server_negotiation.cc:407] Connection header received
0928 02:17:35.340916 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:35.340921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:35.341026 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:35.341151 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:17:35.341824 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:35.343014 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:35.345068 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:35.345723 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:35.345841 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:35.346525 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:17:35.346671 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:17:35.346729 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:35.346952 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 02:17:35.347041 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:17:35.348377 6965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:35.333791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:35461 (local address 127.4.140.195:59633)
0928 02:17:35.340562 (+ 6771us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:35.340587 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:35.340772 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:35.341182 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:35.341187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:35.341200 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:17:35.341626 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:35.341638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:35.343216 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:35.343221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:35.344898 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:35.344917 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:35.346037 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:35.346045 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:35.346253 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:35.346328 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:17:35.346808 (+ 480us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:17:35.346817 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:17:35.347797 (+ 980us) client_negotiation.cc:240] Negotiation successful
0928 02:17:35.347983 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6528,"thread_start_us":7088,"threads_started":1}
I20260928 02:17:35.361246 6698 raft_consensus.cc:1261] T fd7969d6a483430f85133a9ae1d193f5 P afc578d2b43a495382ccde49ffa32ad1 [term 1 FOLLOWER]: Refusing update from remote peer 6ef1f9420dd940e78cee3642e8b8de4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:17:35.361336 6560 raft_consensus.cc:1261] T fd7969d6a483430f85133a9ae1d193f5 P 554900af951f497489f6a9b68f810d55 [term 1 FOLLOWER]: Refusing update from remote peer 6ef1f9420dd940e78cee3642e8b8de4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:17:35.362025 6961 consensus_queue.cc:1035] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:17:35.362500 6908 consensus_queue.cc:1035] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:17:35.411769 6952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:35.405535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:38725 (local address 127.4.140.193:54363)
0928 02:17:35.405733 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:35.405769 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:35.405896 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:35.406488 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:35.406499 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:35.406513 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:17:35.406743 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:35.406758 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:35.407944 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:35.407955 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:35.409183 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:35.409199 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:35.410340 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:35.410346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:35.410553 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:35.410604 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:17:35.411118 (+ 514us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:17:35.411131 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:17:35.411214 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:17:35.411377 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 02:17:35.412508 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:35.405856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:54363 (local address 127.4.140.195:38725)
0928 02:17:35.406019 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:35.406026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:35.406047 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:17:35.406116 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:35.406122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:35.406210 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:35.406323 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:17:35.406938 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:35.407775 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:35.409381 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:35.410090 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:35.410216 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:35.410829 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:17:35.410976 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:17:35.411063 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:35.412196 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0928 02:17:35.412275 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 02:17:35.413578 6935 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:17:37.034112 6398 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:17:37.034368 6398 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:17:37.038543 6398 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e: Sending DeleteTablet for 3 replicas of tablet fd7969d6a483430f85133a9ae1d193f5
I20260928 02:17:37.039274 6535 tablet_service.cc:1514] Processing DeleteTablet for tablet fd7969d6a483430f85133a9ae1d193f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:37 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:34412
I20260928 02:17:37.039423 6674 tablet_service.cc:1514] Processing DeleteTablet for tablet fd7969d6a483430f85133a9ae1d193f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:37 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:60032
I20260928 02:17:37.039718 6812 tablet_service.cc:1514] Processing DeleteTablet for tablet fd7969d6a483430f85133a9ae1d193f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:37 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:44570
I20260928 02:17:37.040843 6993 tablet_replica.cc:330] T fd7969d6a483430f85133a9ae1d193f5 P 554900af951f497489f6a9b68f810d55: stopping tablet replica
I20260928 02:17:37.041180 6993 raft_consensus.cc:2229] T fd7969d6a483430f85133a9ae1d193f5 P 554900af951f497489f6a9b68f810d55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:17:37.041549 6993 raft_consensus.cc:2258] T fd7969d6a483430f85133a9ae1d193f5 P 554900af951f497489f6a9b68f810d55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:37.043071 6993 ts_tablet_manager.cc:1905] T fd7969d6a483430f85133a9ae1d193f5 P 554900af951f497489f6a9b68f810d55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:37.053906 6995 tablet_replica.cc:330] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e: stopping tablet replica
I20260928 02:17:37.053884 6993 ts_tablet_manager.cc:1918] T fd7969d6a483430f85133a9ae1d193f5 P 554900af951f497489f6a9b68f810d55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:17:37.054077 6993 log.cc:1196] T fd7969d6a483430f85133a9ae1d193f5 P 554900af951f497489f6a9b68f810d55: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/fd7969d6a483430f85133a9ae1d193f5
I20260928 02:17:37.054386 6995 raft_consensus.cc:2229] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:17:37.054759 6993 ts_tablet_manager.cc:1939] T fd7969d6a483430f85133a9ae1d193f5 P 554900af951f497489f6a9b68f810d55: Deleting consensus metadata
I20260928 02:17:37.054888 6994 tablet_replica.cc:330] T fd7969d6a483430f85133a9ae1d193f5 P afc578d2b43a495382ccde49ffa32ad1: stopping tablet replica
I20260928 02:17:37.055053 6995 raft_consensus.cc:2258] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:37.055241 6994 raft_consensus.cc:2229] T fd7969d6a483430f85133a9ae1d193f5 P afc578d2b43a495382ccde49ffa32ad1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:17:37.055485 6994 raft_consensus.cc:2258] T fd7969d6a483430f85133a9ae1d193f5 P afc578d2b43a495382ccde49ffa32ad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:37.056087 6995 ts_tablet_manager.cc:1905] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:37.056507 6383 catalog_manager.cc:4841] TS 554900af951f497489f6a9b68f810d55 (127.4.140.193:35461): tablet fd7969d6a483430f85133a9ae1d193f5 (table test-table [id=439aed791aa541c28bb47ec1fa816abe]) successfully deleted
I20260928 02:17:37.056609 6994 ts_tablet_manager.cc:1905] T fd7969d6a483430f85133a9ae1d193f5 P afc578d2b43a495382ccde49ffa32ad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:37.061939 6994 ts_tablet_manager.cc:1918] T fd7969d6a483430f85133a9ae1d193f5 P afc578d2b43a495382ccde49ffa32ad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:17:37.062187 6994 log.cc:1196] T fd7969d6a483430f85133a9ae1d193f5 P afc578d2b43a495382ccde49ffa32ad1: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/fd7969d6a483430f85133a9ae1d193f5
I20260928 02:17:37.062661 6994 ts_tablet_manager.cc:1939] T fd7969d6a483430f85133a9ae1d193f5 P afc578d2b43a495382ccde49ffa32ad1: Deleting consensus metadata
I20260928 02:17:37.063303 6995 ts_tablet_manager.cc:1918] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:17:37.063459 6995 log.cc:1196] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/fd7969d6a483430f85133a9ae1d193f5
I20260928 02:17:37.063984 6995 ts_tablet_manager.cc:1939] T fd7969d6a483430f85133a9ae1d193f5 P 6ef1f9420dd940e78cee3642e8b8de4e: Deleting consensus metadata
I20260928 02:17:37.063979 6384 catalog_manager.cc:4841] TS afc578d2b43a495382ccde49ffa32ad1 (127.4.140.194:37925): tablet fd7969d6a483430f85133a9ae1d193f5 (table test-table [id=439aed791aa541c28bb47ec1fa816abe]) successfully deleted
I20260928 02:17:37.064965 6383 catalog_manager.cc:4841] TS 6ef1f9420dd940e78cee3642e8b8de4e (127.4.140.195:38725): tablet fd7969d6a483430f85133a9ae1d193f5 (table test-table [id=439aed791aa541c28bb47ec1fa816abe]) successfully deleted
Sep 28 02:17:37 dist-test-slave-h37r krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561857, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:17:37 dist-test-slave-h37r krb5kdc[6353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561857, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:37.098896 7003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:37.084514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56556 (local address 127.4.140.254:34297)
0928 02:17:37.085096 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:37.085103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:37.085214 (+ 111us) server_negotiation.cc:407] Connection header received
0928 02:17:37.085443 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:37.085451 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:37.085520 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:37.085639 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:37.086979 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:37.087760 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:37.089079 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:37.089510 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:37.089643 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:37.093597 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:37.093602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:37.093605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:37.093637 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:37.096323 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:37.096973 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:37.096980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:37.096986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:37.097122 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:37.097501 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:37.097507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:37.097510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:37.097779 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:37.098069 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:37.098330 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 02:17:37.098547 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":275,"threads_started":1}
I20260928 02:17:37.102531 6398 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:17:37.103178 6398 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:17:37.110136 6535 tablet_service.cc:1467] Processing CreateTablet for tablet 57cb95f685f941038c0e03aeecc224fb (DEFAULT_TABLE table=test-table [id=17efb1e5902c4ec18fceb307b2af4109]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:37.110394 6674 tablet_service.cc:1467] Processing CreateTablet for tablet 57cb95f685f941038c0e03aeecc224fb (DEFAULT_TABLE table=test-table [id=17efb1e5902c4ec18fceb307b2af4109]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:37.110419 6812 tablet_service.cc:1467] Processing CreateTablet for tablet 57cb95f685f941038c0e03aeecc224fb (DEFAULT_TABLE table=test-table [id=17efb1e5902c4ec18fceb307b2af4109]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:37.110610 6535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57cb95f685f941038c0e03aeecc224fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:37.110824 6674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57cb95f685f941038c0e03aeecc224fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:37.110849 6812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57cb95f685f941038c0e03aeecc224fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:37.115928 7004 tablet_bootstrap.cc:492] T 57cb95f685f941038c0e03aeecc224fb P afc578d2b43a495382ccde49ffa32ad1: Bootstrap starting.
I20260928 02:17:37.117297 7004 tablet_bootstrap.cc:654] T 57cb95f685f941038c0e03aeecc224fb P afc578d2b43a495382ccde49ffa32ad1: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:37.118650 7005 tablet_bootstrap.cc:492] T 57cb95f685f941038c0e03aeecc224fb P 554900af951f497489f6a9b68f810d55: Bootstrap starting.
I20260928 02:17:37.119794 7004 tablet_bootstrap.cc:492] T 57cb95f685f941038c0e03aeecc224fb P afc578d2b43a495382ccde49ffa32ad1: No bootstrap required, opened a new log
I20260928 02:17:37.119980 7006 tablet_bootstrap.cc:492] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e: Bootstrap starting.
I20260928 02:17:37.120055 7004 ts_tablet_manager.cc:1397] T 57cb95f685f941038c0e03aeecc224fb P afc578d2b43a495382ccde49ffa32ad1: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260928 02:17:37.120995 7005 tablet_bootstrap.cc:654] T 57cb95f685f941038c0e03aeecc224fb P 554900af951f497489f6a9b68f810d55: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:37.120890 7004 raft_consensus.cc:348] T 57cb95f685f941038c0e03aeecc224fb P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:37.121163 7004 raft_consensus.cc:374] T 57cb95f685f941038c0e03aeecc224fb P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:37.121258 7004 raft_consensus.cc:729] T 57cb95f685f941038c0e03aeecc224fb P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afc578d2b43a495382ccde49ffa32ad1, State: Initialized, Role: FOLLOWER
I20260928 02:17:37.121513 7004 consensus_queue.cc:260] T 57cb95f685f941038c0e03aeecc224fb P afc578d2b43a495382ccde49ffa32ad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:37.122428 7004 ts_tablet_manager.cc:1428] T 57cb95f685f941038c0e03aeecc224fb P afc578d2b43a495382ccde49ffa32ad1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:17:37.122848 7006 tablet_bootstrap.cc:654] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:37.123925 7005 tablet_bootstrap.cc:492] T 57cb95f685f941038c0e03aeecc224fb P 554900af951f497489f6a9b68f810d55: No bootstrap required, opened a new log
I20260928 02:17:37.124069 7005 ts_tablet_manager.cc:1397] T 57cb95f685f941038c0e03aeecc224fb P 554900af951f497489f6a9b68f810d55: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260928 02:17:37.124853 7006 tablet_bootstrap.cc:492] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e: No bootstrap required, opened a new log
I20260928 02:17:37.125015 7006 ts_tablet_manager.cc:1397] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 02:17:37.124992 7005 raft_consensus.cc:348] T 57cb95f685f941038c0e03aeecc224fb P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:37.125268 7005 raft_consensus.cc:374] T 57cb95f685f941038c0e03aeecc224fb P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:37.125391 7005 raft_consensus.cc:729] T 57cb95f685f941038c0e03aeecc224fb P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554900af951f497489f6a9b68f810d55, State: Initialized, Role: FOLLOWER
I20260928 02:17:37.125625 7005 consensus_queue.cc:260] T 57cb95f685f941038c0e03aeecc224fb P 554900af951f497489f6a9b68f810d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:37.126253 7005 ts_tablet_manager.cc:1428] T 57cb95f685f941038c0e03aeecc224fb P 554900af951f497489f6a9b68f810d55: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:17:37.126667 7006 raft_consensus.cc:348] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:37.126829 7006 raft_consensus.cc:374] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:37.126962 7006 raft_consensus.cc:729] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef1f9420dd940e78cee3642e8b8de4e, State: Initialized, Role: FOLLOWER
I20260928 02:17:37.127213 7006 consensus_queue.cc:260] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:37.127693 7006 ts_tablet_manager.cc:1428] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:17:37.142539 6987 raft_consensus.cc:482] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:17:37.142756 6987 raft_consensus.cc:504] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:37.143503 6987 leader_election.cc:288] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 554900af951f497489f6a9b68f810d55 (127.4.140.193:35461), afc578d2b43a495382ccde49ffa32ad1 (127.4.140.194:37925)
I20260928 02:17:37.144037 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57cb95f685f941038c0e03aeecc224fb" candidate_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554900af951f497489f6a9b68f810d55" is_pre_election: true
I20260928 02:17:37.144091 6698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57cb95f685f941038c0e03aeecc224fb" candidate_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afc578d2b43a495382ccde49ffa32ad1" is_pre_election: true
I20260928 02:17:37.144306 6698 raft_consensus.cc:2454] T 57cb95f685f941038c0e03aeecc224fb P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ef1f9420dd940e78cee3642e8b8de4e in term 0.
I20260928 02:17:37.144306 6559 raft_consensus.cc:2454] T 57cb95f685f941038c0e03aeecc224fb P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ef1f9420dd940e78cee3642e8b8de4e in term 0.
I20260928 02:17:37.144796 6765 leader_election.cc:302] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 554900af951f497489f6a9b68f810d55, 6ef1f9420dd940e78cee3642e8b8de4e; no voters:
I20260928 02:17:37.145047 6987 raft_consensus.cc:2781] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:17:37.145186 6987 raft_consensus.cc:482] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:17:37.145315 6987 raft_consensus.cc:3037] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:37.147641 6987 raft_consensus.cc:504] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:37.148424 6987 leader_election.cc:288] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [CANDIDATE]: Term 1 election: Requested vote from peers 554900af951f497489f6a9b68f810d55 (127.4.140.193:35461), afc578d2b43a495382ccde49ffa32ad1 (127.4.140.194:37925)
I20260928 02:17:37.148871 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57cb95f685f941038c0e03aeecc224fb" candidate_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554900af951f497489f6a9b68f810d55"
I20260928 02:17:37.148909 6698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57cb95f685f941038c0e03aeecc224fb" candidate_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afc578d2b43a495382ccde49ffa32ad1"
I20260928 02:17:37.149113 6698 raft_consensus.cc:3037] T 57cb95f685f941038c0e03aeecc224fb P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:37.149139 6559 raft_consensus.cc:3037] T 57cb95f685f941038c0e03aeecc224fb P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:37.151471 6698 raft_consensus.cc:2454] T 57cb95f685f941038c0e03aeecc224fb P afc578d2b43a495382ccde49ffa32ad1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef1f9420dd940e78cee3642e8b8de4e in term 1.
I20260928 02:17:37.151475 6559 raft_consensus.cc:2454] T 57cb95f685f941038c0e03aeecc224fb P 554900af951f497489f6a9b68f810d55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef1f9420dd940e78cee3642e8b8de4e in term 1.
I20260928 02:17:37.151976 6765 leader_election.cc:302] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 554900af951f497489f6a9b68f810d55, 6ef1f9420dd940e78cee3642e8b8de4e; no voters:
I20260928 02:17:37.152325 6987 raft_consensus.cc:2781] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:37.152508 6987 raft_consensus.cc:686] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 LEADER]: Becoming Leader. State: Replica: 6ef1f9420dd940e78cee3642e8b8de4e, State: Running, Role: LEADER
I20260928 02:17:37.152834 6987 consensus_queue.cc:237] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:37.155459 6398 catalog_manager.cc:5495] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ef1f9420dd940e78cee3642e8b8de4e (127.4.140.195). New cstate: current_term: 1 leader_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } health_report { overall_health: HEALTHY } } }
I20260928 02:17:37.186170 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:37.181120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48828 (local address 127.4.140.195:38725)
0928 02:17:37.181723 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:37.181727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:37.181745 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:17:37.181793 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:37.181797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:37.181858 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:37.182000 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:17:37.182305 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:37.183133 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:37.184589 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:37.185004 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:37.185098 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:37.185459 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:17:37.185560 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:17:37.185626 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:37.185743 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 02:17:37.185797 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":382,"threads_started":1}
I20260928 02:17:37.194609 6559 raft_consensus.cc:1261] T 57cb95f685f941038c0e03aeecc224fb P 554900af951f497489f6a9b68f810d55 [term 1 FOLLOWER]: Refusing update from remote peer 6ef1f9420dd940e78cee3642e8b8de4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:17:37.194607 6698 raft_consensus.cc:1261] T 57cb95f685f941038c0e03aeecc224fb P afc578d2b43a495382ccde49ffa32ad1 [term 1 FOLLOWER]: Refusing update from remote peer 6ef1f9420dd940e78cee3642e8b8de4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:17:37.195137 6987 consensus_queue.cc:1035] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:17:37.195446 6970 consensus_queue.cc:1035] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:17:37.238467 6396 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:17:37.238729 6396 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:17:37.241700 6396 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e: Sending DeleteTablet for 3 replicas of tablet 57cb95f685f941038c0e03aeecc224fb
I20260928 02:17:37.242527 6535 tablet_service.cc:1514] Processing DeleteTablet for tablet 57cb95f685f941038c0e03aeecc224fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:37 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:34412
I20260928 02:17:37.242713 6674 tablet_service.cc:1514] Processing DeleteTablet for tablet 57cb95f685f941038c0e03aeecc224fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:37 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:60032
I20260928 02:17:37.242751 6812 tablet_service.cc:1514] Processing DeleteTablet for tablet 57cb95f685f941038c0e03aeecc224fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:37 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:44570
I20260928 02:17:37.242980 6993 tablet_replica.cc:330] T 57cb95f685f941038c0e03aeecc224fb P 554900af951f497489f6a9b68f810d55: stopping tablet replica
I20260928 02:17:37.242991 6995 tablet_replica.cc:330] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e: stopping tablet replica
I20260928 02:17:37.243000 6994 tablet_replica.cc:330] T 57cb95f685f941038c0e03aeecc224fb P afc578d2b43a495382ccde49ffa32ad1: stopping tablet replica
I20260928 02:17:37.243144 6994 raft_consensus.cc:2229] T 57cb95f685f941038c0e03aeecc224fb P afc578d2b43a495382ccde49ffa32ad1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:17:37.243172 6993 raft_consensus.cc:2229] T 57cb95f685f941038c0e03aeecc224fb P 554900af951f497489f6a9b68f810d55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:17:37.243175 6995 raft_consensus.cc:2229] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:17:37.243278 6994 raft_consensus.cc:2258] T 57cb95f685f941038c0e03aeecc224fb P afc578d2b43a495382ccde49ffa32ad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:37.243340 6993 raft_consensus.cc:2258] T 57cb95f685f941038c0e03aeecc224fb P 554900af951f497489f6a9b68f810d55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:37.243521 6995 raft_consensus.cc:2258] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:37.244313 6994 ts_tablet_manager.cc:1905] T 57cb95f685f941038c0e03aeecc224fb P afc578d2b43a495382ccde49ffa32ad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:37.244536 6995 ts_tablet_manager.cc:1905] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:37.245033 6993 ts_tablet_manager.cc:1905] T 57cb95f685f941038c0e03aeecc224fb P 554900af951f497489f6a9b68f810d55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:37.249123 6396 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:17:37.249634 6994 ts_tablet_manager.cc:1918] T 57cb95f685f941038c0e03aeecc224fb P afc578d2b43a495382ccde49ffa32ad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:17:37.250016 6994 log.cc:1196] T 57cb95f685f941038c0e03aeecc224fb P afc578d2b43a495382ccde49ffa32ad1: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/57cb95f685f941038c0e03aeecc224fb
W20260928 02:17:37.250089 6396 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:17:37.250818 6994 ts_tablet_manager.cc:1939] T 57cb95f685f941038c0e03aeecc224fb P afc578d2b43a495382ccde49ffa32ad1: Deleting consensus metadata
I20260928 02:17:37.251427 6993 ts_tablet_manager.cc:1918] T 57cb95f685f941038c0e03aeecc224fb P 554900af951f497489f6a9b68f810d55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:17:37.251646 6993 log.cc:1196] T 57cb95f685f941038c0e03aeecc224fb P 554900af951f497489f6a9b68f810d55: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/57cb95f685f941038c0e03aeecc224fb
I20260928 02:17:37.252704 6993 ts_tablet_manager.cc:1939] T 57cb95f685f941038c0e03aeecc224fb P 554900af951f497489f6a9b68f810d55: Deleting consensus metadata
I20260928 02:17:37.252969 6384 catalog_manager.cc:4841] TS afc578d2b43a495382ccde49ffa32ad1 (127.4.140.194:37925): tablet 57cb95f685f941038c0e03aeecc224fb (table test-table [id=17efb1e5902c4ec18fceb307b2af4109]) successfully deleted
I20260928 02:17:37.253806 6383 catalog_manager.cc:4841] TS 554900af951f497489f6a9b68f810d55 (127.4.140.193:35461): tablet 57cb95f685f941038c0e03aeecc224fb (table test-table [id=17efb1e5902c4ec18fceb307b2af4109]) successfully deleted
I20260928 02:17:37.254127 6995 ts_tablet_manager.cc:1918] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:17:37.254287 6995 log.cc:1196] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/57cb95f685f941038c0e03aeecc224fb
I20260928 02:17:37.254667 6995 ts_tablet_manager.cc:1939] T 57cb95f685f941038c0e03aeecc224fb P 6ef1f9420dd940e78cee3642e8b8de4e: Deleting consensus metadata
I20260928 02:17:37.255693 6383 catalog_manager.cc:4841] TS 6ef1f9420dd940e78cee3642e8b8de4e (127.4.140.195:38725): tablet 57cb95f685f941038c0e03aeecc224fb (table test-table [id=17efb1e5902c4ec18fceb307b2af4109]) successfully deleted
I20260928 02:17:37.257805 6535 tablet_service.cc:1467] Processing CreateTablet for tablet 84ede8ac52e649e0891eb5be42eab4d5 (DEFAULT_TABLE table=test-table [id=4330753293ca4523b2e1db1c1bb406c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:37.258081 6812 tablet_service.cc:1467] Processing CreateTablet for tablet 84ede8ac52e649e0891eb5be42eab4d5 (DEFAULT_TABLE table=test-table [id=4330753293ca4523b2e1db1c1bb406c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:37.258095 6674 tablet_service.cc:1467] Processing CreateTablet for tablet 84ede8ac52e649e0891eb5be42eab4d5 (DEFAULT_TABLE table=test-table [id=4330753293ca4523b2e1db1c1bb406c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:37.258255 6535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ede8ac52e649e0891eb5be42eab4d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:37.258448 6812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ede8ac52e649e0891eb5be42eab4d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:37.258556 6674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ede8ac52e649e0891eb5be42eab4d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:37.263437 7004 tablet_bootstrap.cc:492] T 84ede8ac52e649e0891eb5be42eab4d5 P afc578d2b43a495382ccde49ffa32ad1: Bootstrap starting.
I20260928 02:17:37.264595 7005 tablet_bootstrap.cc:492] T 84ede8ac52e649e0891eb5be42eab4d5 P 554900af951f497489f6a9b68f810d55: Bootstrap starting.
I20260928 02:17:37.264962 7004 tablet_bootstrap.cc:654] T 84ede8ac52e649e0891eb5be42eab4d5 P afc578d2b43a495382ccde49ffa32ad1: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:37.266253 7005 tablet_bootstrap.cc:654] T 84ede8ac52e649e0891eb5be42eab4d5 P 554900af951f497489f6a9b68f810d55: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:37.267067 7004 tablet_bootstrap.cc:492] T 84ede8ac52e649e0891eb5be42eab4d5 P afc578d2b43a495382ccde49ffa32ad1: No bootstrap required, opened a new log
I20260928 02:17:37.267241 7006 tablet_bootstrap.cc:492] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e: Bootstrap starting.
I20260928 02:17:37.267252 7004 ts_tablet_manager.cc:1397] T 84ede8ac52e649e0891eb5be42eab4d5 P afc578d2b43a495382ccde49ffa32ad1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 02:17:37.267839 7004 raft_consensus.cc:348] T 84ede8ac52e649e0891eb5be42eab4d5 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:37.268136 7004 raft_consensus.cc:374] T 84ede8ac52e649e0891eb5be42eab4d5 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:37.268247 7004 raft_consensus.cc:729] T 84ede8ac52e649e0891eb5be42eab4d5 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afc578d2b43a495382ccde49ffa32ad1, State: Initialized, Role: FOLLOWER
I20260928 02:17:37.268507 7004 consensus_queue.cc:260] T 84ede8ac52e649e0891eb5be42eab4d5 P afc578d2b43a495382ccde49ffa32ad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:37.268640 7006 tablet_bootstrap.cc:654] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:37.268990 7004 ts_tablet_manager.cc:1428] T 84ede8ac52e649e0891eb5be42eab4d5 P afc578d2b43a495382ccde49ffa32ad1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:17:37.270478 7005 tablet_bootstrap.cc:492] T 84ede8ac52e649e0891eb5be42eab4d5 P 554900af951f497489f6a9b68f810d55: No bootstrap required, opened a new log
I20260928 02:17:37.270643 7005 ts_tablet_manager.cc:1397] T 84ede8ac52e649e0891eb5be42eab4d5 P 554900af951f497489f6a9b68f810d55: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260928 02:17:37.271322 7005 raft_consensus.cc:348] T 84ede8ac52e649e0891eb5be42eab4d5 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:37.271534 7005 raft_consensus.cc:374] T 84ede8ac52e649e0891eb5be42eab4d5 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:37.271637 7005 raft_consensus.cc:729] T 84ede8ac52e649e0891eb5be42eab4d5 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554900af951f497489f6a9b68f810d55, State: Initialized, Role: FOLLOWER
I20260928 02:17:37.272015 7005 consensus_queue.cc:260] T 84ede8ac52e649e0891eb5be42eab4d5 P 554900af951f497489f6a9b68f810d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:37.272541 7005 ts_tablet_manager.cc:1428] T 84ede8ac52e649e0891eb5be42eab4d5 P 554900af951f497489f6a9b68f810d55: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:17:37.280514 7006 tablet_bootstrap.cc:492] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e: No bootstrap required, opened a new log
I20260928 02:17:37.280730 7006 ts_tablet_manager.cc:1397] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20260928 02:17:37.281497 7006 raft_consensus.cc:348] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:37.281709 7006 raft_consensus.cc:374] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:37.281816 7006 raft_consensus.cc:729] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef1f9420dd940e78cee3642e8b8de4e, State: Initialized, Role: FOLLOWER
I20260928 02:17:37.282095 7006 consensus_queue.cc:260] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:37.282590 7006 ts_tablet_manager.cc:1428] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:17:37.362844 6987 raft_consensus.cc:482] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:17:37.363058 6987 raft_consensus.cc:504] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:37.364023 6987 leader_election.cc:288] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 554900af951f497489f6a9b68f810d55 (127.4.140.193:35461), afc578d2b43a495382ccde49ffa32ad1 (127.4.140.194:37925)
I20260928 02:17:37.364645 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ede8ac52e649e0891eb5be42eab4d5" candidate_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554900af951f497489f6a9b68f810d55" is_pre_election: true
I20260928 02:17:37.364776 6698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ede8ac52e649e0891eb5be42eab4d5" candidate_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afc578d2b43a495382ccde49ffa32ad1" is_pre_election: true
I20260928 02:17:37.364921 6559 raft_consensus.cc:2454] T 84ede8ac52e649e0891eb5be42eab4d5 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ef1f9420dd940e78cee3642e8b8de4e in term 0.
I20260928 02:17:37.365094 6698 raft_consensus.cc:2454] T 84ede8ac52e649e0891eb5be42eab4d5 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ef1f9420dd940e78cee3642e8b8de4e in term 0.
I20260928 02:17:37.365381 6765 leader_election.cc:302] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 554900af951f497489f6a9b68f810d55, 6ef1f9420dd940e78cee3642e8b8de4e; no voters:
I20260928 02:17:37.365720 6987 raft_consensus.cc:2781] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:17:37.365852 6987 raft_consensus.cc:482] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:17:37.365988 6987 raft_consensus.cc:3037] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:37.368396 6987 raft_consensus.cc:504] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:37.369055 6987 leader_election.cc:288] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [CANDIDATE]: Term 1 election: Requested vote from peers 554900af951f497489f6a9b68f810d55 (127.4.140.193:35461), afc578d2b43a495382ccde49ffa32ad1 (127.4.140.194:37925)
I20260928 02:17:37.369612 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ede8ac52e649e0891eb5be42eab4d5" candidate_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554900af951f497489f6a9b68f810d55"
I20260928 02:17:37.369683 6698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ede8ac52e649e0891eb5be42eab4d5" candidate_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afc578d2b43a495382ccde49ffa32ad1"
I20260928 02:17:37.369832 6559 raft_consensus.cc:3037] T 84ede8ac52e649e0891eb5be42eab4d5 P 554900af951f497489f6a9b68f810d55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:37.369973 6698 raft_consensus.cc:3037] T 84ede8ac52e649e0891eb5be42eab4d5 P afc578d2b43a495382ccde49ffa32ad1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:37.372669 6559 raft_consensus.cc:2454] T 84ede8ac52e649e0891eb5be42eab4d5 P 554900af951f497489f6a9b68f810d55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef1f9420dd940e78cee3642e8b8de4e in term 1.
I20260928 02:17:37.373051 6698 raft_consensus.cc:2454] T 84ede8ac52e649e0891eb5be42eab4d5 P afc578d2b43a495382ccde49ffa32ad1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef1f9420dd940e78cee3642e8b8de4e in term 1.
I20260928 02:17:37.373222 6765 leader_election.cc:302] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 554900af951f497489f6a9b68f810d55, 6ef1f9420dd940e78cee3642e8b8de4e; no voters:
I20260928 02:17:37.373557 6987 raft_consensus.cc:2781] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:37.373713 6987 raft_consensus.cc:686] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 LEADER]: Becoming Leader. State: Replica: 6ef1f9420dd940e78cee3642e8b8de4e, State: Running, Role: LEADER
I20260928 02:17:37.373975 6987 consensus_queue.cc:237] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } }
I20260928 02:17:37.376632 6398 catalog_manager.cc:5495] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ef1f9420dd940e78cee3642e8b8de4e (127.4.140.195). New cstate: current_term: 1 leader_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ef1f9420dd940e78cee3642e8b8de4e" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38725 } health_report { overall_health: HEALTHY } } }
I20260928 02:17:37.409900 6698 raft_consensus.cc:1261] T 84ede8ac52e649e0891eb5be42eab4d5 P afc578d2b43a495382ccde49ffa32ad1 [term 1 FOLLOWER]: Refusing update from remote peer 6ef1f9420dd940e78cee3642e8b8de4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:17:37.410024 6559 raft_consensus.cc:1261] T 84ede8ac52e649e0891eb5be42eab4d5 P 554900af951f497489f6a9b68f810d55 [term 1 FOLLOWER]: Refusing update from remote peer 6ef1f9420dd940e78cee3642e8b8de4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:17:37.410557 6985 consensus_queue.cc:1035] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "afc578d2b43a495382ccde49ffa32ad1" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:17:37.410874 6987 consensus_queue.cc:1035] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "554900af951f497489f6a9b68f810d55" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:17:37.430289 6974 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:17:39.014027 6398 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:17:39.014386 6398 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:17:39.018805 6398 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6c786e2fd86435ebfd299ea6bed2b7e: Sending DeleteTablet for 3 replicas of tablet 84ede8ac52e649e0891eb5be42eab4d5
I20260928 02:17:39.019798 6535 tablet_service.cc:1514] Processing DeleteTablet for tablet 84ede8ac52e649e0891eb5be42eab4d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:39 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:34412
I20260928 02:17:39.019840 6812 tablet_service.cc:1514] Processing DeleteTablet for tablet 84ede8ac52e649e0891eb5be42eab4d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:39 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:44570
I20260928 02:17:39.019981 6674 tablet_service.cc:1514] Processing DeleteTablet for tablet 84ede8ac52e649e0891eb5be42eab4d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:39 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:60032
I20260928 02:17:39.020538 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 6469
I20260928 02:17:39.023247 7045 tablet_replica.cc:330] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e: stopping tablet replica
I20260928 02:17:39.023470 7044 tablet_replica.cc:330] T 84ede8ac52e649e0891eb5be42eab4d5 P 554900af951f497489f6a9b68f810d55: stopping tablet replica
I20260928 02:17:39.023526 7045 raft_consensus.cc:2229] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:17:39.023721 7044 raft_consensus.cc:2229] T 84ede8ac52e649e0891eb5be42eab4d5 P 554900af951f497489f6a9b68f810d55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:17:39.023957 7045 raft_consensus.cc:2258] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:39.024021 7044 raft_consensus.cc:2258] T 84ede8ac52e649e0891eb5be42eab4d5 P 554900af951f497489f6a9b68f810d55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:39.024593 7046 tablet_replica.cc:330] T 84ede8ac52e649e0891eb5be42eab4d5 P afc578d2b43a495382ccde49ffa32ad1: stopping tablet replica
I20260928 02:17:39.024833 7046 raft_consensus.cc:2229] T 84ede8ac52e649e0891eb5be42eab4d5 P afc578d2b43a495382ccde49ffa32ad1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:17:39.025084 7045 ts_tablet_manager.cc:1905] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:39.025108 7046 raft_consensus.cc:2258] T 84ede8ac52e649e0891eb5be42eab4d5 P afc578d2b43a495382ccde49ffa32ad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:39.025822 7044 ts_tablet_manager.cc:1905] T 84ede8ac52e649e0891eb5be42eab4d5 P 554900af951f497489f6a9b68f810d55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:39.026391 7046 ts_tablet_manager.cc:1905] T 84ede8ac52e649e0891eb5be42eab4d5 P afc578d2b43a495382ccde49ffa32ad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:39.032462 7045 ts_tablet_manager.cc:1918] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:17:39.032773 7046 ts_tablet_manager.cc:1918] T 84ede8ac52e649e0891eb5be42eab4d5 P afc578d2b43a495382ccde49ffa32ad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:17:39.033334 7046 log.cc:1196] T 84ede8ac52e649e0891eb5be42eab4d5 P afc578d2b43a495382ccde49ffa32ad1: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/84ede8ac52e649e0891eb5be42eab4d5
I20260928 02:17:39.033702 7045 log.cc:1196] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/84ede8ac52e649e0891eb5be42eab4d5
I20260928 02:17:39.033892 7046 ts_tablet_manager.cc:1939] T 84ede8ac52e649e0891eb5be42eab4d5 P afc578d2b43a495382ccde49ffa32ad1: Deleting consensus metadata
I20260928 02:17:39.034247 7045 ts_tablet_manager.cc:1939] T 84ede8ac52e649e0891eb5be42eab4d5 P 6ef1f9420dd940e78cee3642e8b8de4e: Deleting consensus metadata
I20260928 02:17:39.035055 6384 catalog_manager.cc:4841] TS afc578d2b43a495382ccde49ffa32ad1 (127.4.140.194:37925): tablet 84ede8ac52e649e0891eb5be42eab4d5 (table test-table [id=4330753293ca4523b2e1db1c1bb406c7]) successfully deleted
I20260928 02:17:39.035527 6383 catalog_manager.cc:4841] TS 6ef1f9420dd940e78cee3642e8b8de4e (127.4.140.195:38725): tablet 84ede8ac52e649e0891eb5be42eab4d5 (table test-table [id=4330753293ca4523b2e1db1c1bb406c7]) successfully deleted
I20260928 02:17:39.038388 7044 ts_tablet_manager.cc:1918] T 84ede8ac52e649e0891eb5be42eab4d5 P 554900af951f497489f6a9b68f810d55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:17:39.038568 7044 log.cc:1196] T 84ede8ac52e649e0891eb5be42eab4d5 P 554900af951f497489f6a9b68f810d55: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/84ede8ac52e649e0891eb5be42eab4d5
I20260928 02:17:39.039031 7044 ts_tablet_manager.cc:1939] T 84ede8ac52e649e0891eb5be42eab4d5 P 554900af951f497489f6a9b68f810d55: Deleting consensus metadata
I20260928 02:17:39.040040 6383 catalog_manager.cc:4841] TS 554900af951f497489f6a9b68f810d55 (127.4.140.193:35461): tablet 84ede8ac52e649e0891eb5be42eab4d5 (table test-table [id=4330753293ca4523b2e1db1c1bb406c7]) successfully deleted
Sep 28 02:17:39 dist-test-slave-h37r krb5kdc[6353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561857, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:17:39.062516 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:39.021550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45326 (local address 127.4.140.193:35461)
0928 02:17:39.033225 (+ 11675us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:39.033232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:39.033562 (+ 330us) server_negotiation.cc:407] Connection header received
0928 02:17:39.033818 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:39.033823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:39.033925 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:39.034052 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:39.035946 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:39.036790 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:39.038095 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:39.038627 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:39.038790 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:39.056629 (+ 17839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:39.056647 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:39.056664 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:39.056706 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:39.059537 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:39.060489 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:39.060498 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:39.060500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:39.060652 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:39.061146 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:39.061150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:39.061152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:39.061513 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:39.061747 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:39.062070 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 02:17:39.062243 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":11512,"thread_start_us":11425,"threads_started":1}
W20260928 02:17:39.063287 6600 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45326
W20260928 02:17:39.063999 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 6469: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:17:39.064167 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 6469
I20260928 02:17:39.086675 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 6609
Sep 28 02:17:39 dist-test-slave-h37r krb5kdc[6353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561857, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.140.194@KRBTEST.COM
I20260928 02:17:39.102299 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:39.087352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42488 (local address 127.4.140.194:37925)
0928 02:17:39.087853 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:39.087857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:39.087873 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:17:39.088015 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:39.088019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:39.088082 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:39.088172 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:39.089530 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:39.090259 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:39.091491 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:39.091852 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:39.092031 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:39.096182 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:39.096196 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:39.096211 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:39.096250 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:39.099514 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:39.100381 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:39.100390 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:39.100392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:39.100525 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:39.100954 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:39.100960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:39.100962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:39.101322 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:39.101539 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:39.101836 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 02:17:39.102020 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":266,"threads_started":1}
W20260928 02:17:39.103160 6739 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42488
W20260928 02:17:39.103781 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 6609: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:17:39.103979 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 6609
I20260928 02:17:39.125631 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 6748
Sep 28 02:17:39 dist-test-slave-h37r krb5kdc[6353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561857, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.140.195@KRBTEST.COM
I20260928 02:17:39.141346 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:39.126351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48838 (local address 127.4.140.195:38725)
0928 02:17:39.126856 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:39.126861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:39.126876 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:17:39.126928 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:39.126932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:39.126993 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:39.127083 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:39.128459 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:39.129244 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:39.130486 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:39.130882 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:39.130980 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:39.135273 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:39.135290 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:39.135307 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:39.135346 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:39.138321 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:39.139039 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:39.139051 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:39.139055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:39.139195 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:39.139635 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:39.139639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:39.139641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:39.140257 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:39.140509 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:39.140825 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 02:17:39.141076 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":267,"threads_started":1}
W20260928 02:17:39.142092 6878 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48838
W20260928 02:17:39.142925 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 6748: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:17:39.143085 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 6748
I20260928 02:17:39.164790 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 6366
I20260928 02:17:39.165964 6438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:39.282830 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 6366
2026-09-28T02:17:39Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8693 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:17:39 dist-test-slave-h37r krb5kdc[7054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:17:39 dist-test-slave-h37r krb5kdc[7054](info): set up 2 sockets
Sep 28 02:17:39 dist-test-slave-h37r krb5kdc[7054](info): commencing operation
krb5kdc: starting...
W20260928 02:17:40.368686 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.008s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:17:40 dist-test-slave-h37r krb5kdc[7054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:17:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:17:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:17:40.559158 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:32827
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:32827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:40.898355 7067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:40.898645 7067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:40.898806 7067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:40.906311 7067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:17:40.906430 7067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:40.906512 7067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:40.906596 7067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:17:40.906697 7067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:17:40.917429 7067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:32827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:32827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:40.918805 7067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:40.920331 7067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:40.933576 7072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:40.933650 7073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:40.933629 7075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:40.934038 7067 server_base.cc:1034] running on GCE node
I20260928 02:17:40.934847 7067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:40.936347 7067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:40.937598 7067 hybrid_clock.cc:648] HybridClock initialized: now 1790561860937540 us; error 63 us; skew 500 ppm
I20260928 02:17:40.937997 7067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:40 dist-test-slave-h37r krb5kdc[7054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561860, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:40.941237 7067 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:17:40.942699 7067 webserver.cc:460] Webserver started at http://127.4.140.254:34155/ using document root <none> and password file <none>
I20260928 02:17:40.943403 7067 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:40.943545 7067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:40.943862 7067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:40.946791 7067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "ecd45bf29f9149ce994ff4fe70c818ed"
format_stamp: "Formatted at 2026-09-28 02:17:40 on dist-test-slave-h37r"
server_key: "b43b9d9d149d3497fc38e07be36ced39"
server_key_iv: "8f2b47b90db908921d152a0b19be4def"
server_key_version: "encryptionkey@0"
I20260928 02:17:40.947556 7067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "ecd45bf29f9149ce994ff4fe70c818ed"
format_stamp: "Formatted at 2026-09-28 02:17:40 on dist-test-slave-h37r"
server_key: "b43b9d9d149d3497fc38e07be36ced39"
server_key_iv: "8f2b47b90db908921d152a0b19be4def"
server_key_version: "encryptionkey@0"
I20260928 02:17:40.952690 7067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260928 02:17:40.956278 7082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:40.957617 7067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 02:17:40.957823 7067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "ecd45bf29f9149ce994ff4fe70c818ed"
format_stamp: "Formatted at 2026-09-28 02:17:40 on dist-test-slave-h37r"
server_key: "b43b9d9d149d3497fc38e07be36ced39"
server_key_iv: "8f2b47b90db908921d152a0b19be4def"
server_key_version: "encryptionkey@0"
I20260928 02:17:40.958015 7067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:40.977952 7067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:40.981989 7067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:40.982285 7067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:41.001194 7067 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:32827
I20260928 02:17:41.001223 7133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:32827 every 8 connection(s)
I20260928 02:17:41.002580 7067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:17:41.006346 7134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:41.007150 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 7067
I20260928 02:17:41.007359 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:17:41.008051 4659 external_mini_cluster.cc:1428] Setting key 9e11b7b73eb71ebdd612ca51c946c713
I20260928 02:17:41.020474 7134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed: Bootstrap starting.
Sep 28 02:17:41 dist-test-slave-h37r krb5kdc[7054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:41.024209 7134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:41.025243 7134 log.cc:824] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:41.028189 7134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed: No bootstrap required, opened a new log
I20260928 02:17:41.031486 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:41.010016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34846 (local address 127.4.140.254:32827)
0928 02:17:41.010787 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:41.010800 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:41.010843 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:17:41.012498 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:41.012524 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:41.012923 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:41.013252 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:41.015107 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:41.016641 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:41.018509 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:41.019196 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:41.019345 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:41.024011 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:41.024020 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:41.024034 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:41.024074 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:41.027523 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:41.028317 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:41.028327 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:41.028335 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:41.028514 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:41.028880 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:41.028887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:41.028890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:41.029360 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:41.029675 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:41.030127 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 02:17:41.030524 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":352,"threads_started":1}
I20260928 02:17:41.036039 7134 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd45bf29f9149ce994ff4fe70c818ed" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32827 } }
I20260928 02:17:41.036399 7134 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:41.036523 7134 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecd45bf29f9149ce994ff4fe70c818ed, State: Initialized, Role: FOLLOWER
I20260928 02:17:41.037132 7134 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd45bf29f9149ce994ff4fe70c818ed" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32827 } }
I20260928 02:17:41.037364 7134 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:17:41.037532 7134 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:17:41.037709 7134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:41.040148 7134 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd45bf29f9149ce994ff4fe70c818ed" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32827 } }
I20260928 02:17:41.040642 7134 leader_election.cc:302] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecd45bf29f9149ce994ff4fe70c818ed; no voters:
I20260928 02:17:41.041208 7134 leader_election.cc:288] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:17:41.041332 7139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:41.042073 7139 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed [term 1 LEADER]: Becoming Leader. State: Replica: ecd45bf29f9149ce994ff4fe70c818ed, State: Running, Role: LEADER
I20260928 02:17:41.042662 7139 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd45bf29f9149ce994ff4fe70c818ed" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32827 } }
I20260928 02:17:41.043208 7134 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:17:41.045253 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecd45bf29f9149ce994ff4fe70c818ed. Latest consensus state: current_term: 1 leader_uuid: "ecd45bf29f9149ce994ff4fe70c818ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd45bf29f9149ce994ff4fe70c818ed" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32827 } } }
I20260928 02:17:41.045598 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:41.046443 7144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:17:41.046324 7140 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecd45bf29f9149ce994ff4fe70c818ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd45bf29f9149ce994ff4fe70c818ed" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32827 } } }
I20260928 02:17:41.047590 7140 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:41.051977 7144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:17:41.063279 7144 catalog_manager.cc:1347] Generated new cluster ID: bea6c26cfa654dc483e49f0c5ee777b1
I20260928 02:17:41.063408 7144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:17:41.082346 7144 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:17:41.083679 7144 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:17:41.096565 7144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecd45bf29f9149ce994ff4fe70c818ed: Generated new TSK 0
I20260928 02:17:41.097819 7144 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
I20260928 02:17:41.151832 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:32827
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:41.495254 7160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:41.495537 7160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:41.495687 7160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:41.503090 7160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:41.503201 7160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:41.503381 7160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:17:41.513288 7160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:32827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:41.514616 7160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:41.516019 7160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:41.529414 7165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:41.529444 7166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:41.529381 7168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:41.530298 7160 server_base.cc:1034] running on GCE node
I20260928 02:17:41.530956 7160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:41.532099 7160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:41.533349 7160 hybrid_clock.cc:648] HybridClock initialized: now 1790561861533277 us; error 64 us; skew 500 ppm
I20260928 02:17:41.533754 7160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:41 dist-test-slave-h37r krb5kdc[7054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561861, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:41.537075 7160 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:17:41.538566 7160 webserver.cc:460] Webserver started at http://127.4.140.193:34387/ using document root <none> and password file <none>
I20260928 02:17:41.539281 7160 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:41.539433 7160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:41.539757 7160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:41.542837 7160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "34b26fef40064ab2875bf65ce8bfdb04"
format_stamp: "Formatted at 2026-09-28 02:17:41 on dist-test-slave-h37r"
server_key: "e011478ff6c53ea46e21d4ce19f6fa3b"
server_key_iv: "8c278a571c1eb7f0e13ceba33f7d3bc4"
server_key_version: "encryptionkey@0"
I20260928 02:17:41.543738 7160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "34b26fef40064ab2875bf65ce8bfdb04"
format_stamp: "Formatted at 2026-09-28 02:17:41 on dist-test-slave-h37r"
server_key: "e011478ff6c53ea46e21d4ce19f6fa3b"
server_key_iv: "8c278a571c1eb7f0e13ceba33f7d3bc4"
server_key_version: "encryptionkey@0"
I20260928 02:17:41.549216 7160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 02:17:41.553053 7175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:41.554593 7160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:17:41.554759 7160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "34b26fef40064ab2875bf65ce8bfdb04"
format_stamp: "Formatted at 2026-09-28 02:17:41 on dist-test-slave-h37r"
server_key: "e011478ff6c53ea46e21d4ce19f6fa3b"
server_key_iv: "8c278a571c1eb7f0e13ceba33f7d3bc4"
server_key_version: "encryptionkey@0"
I20260928 02:17:41.554961 7160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:41.572876 7160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:41.577028 7160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:41.577351 7160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:41.578301 7160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:41.579623 7160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:41.579694 7160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:41.579851 7160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:41.579952 7160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 02:17:41.613015 7160 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:45301
I20260928 02:17:41.613059 7287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:45301 every 8 connection(s)
I20260928 02:17:41.614279 7160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:17:41.623592 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 7160
I20260928 02:17:41.623812 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:17:41.624675 4659 external_mini_cluster.cc:1428] Setting key ca3b6da5dcef148e440bfee433dcd011
Sep 28 02:17:41 dist-test-slave-h37r krb5kdc[7054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561861, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:41.635701 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:41.616922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:54181 (local address 127.4.140.254:32827)
0928 02:17:41.617315 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:41.617319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:41.618350 (+ 1031us) server_negotiation.cc:407] Connection header received
0928 02:17:41.620125 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:41.620129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:41.620192 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:41.620285 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:41.621707 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:41.622392 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:41.624043 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:41.624704 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:41.624856 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:41.628676 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:41.628685 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:41.628692 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:41.628734 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:41.632240 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:41.633241 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:41.633245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:41.633247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:41.633360 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:41.633892 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:41.633896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:41.633898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:41.634181 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:41.634320 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:41.635269 (+ 949us) server_negotiation.cc:299] Negotiation successful
0928 02:17:41.635447 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":186,"threads_started":1}
I20260928 02:17:41.637310 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:41.617510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32827 (local address 127.4.140.193:54181)
0928 02:17:41.618180 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:41.618225 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:41.619825 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:41.620485 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:41.620492 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:41.621011 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:41.621557 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:41.621574 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:41.622548 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:41.622552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:41.623856 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:41.623868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:41.624931 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:41.624936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:41.625196 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:41.625985 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:41.626022 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:41.628447 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:41.632411 (+ 3964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:41.632423 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:41.632444 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:41.633100 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:41.633477 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:41.633483 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:41.633486 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:41.633739 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:41.634313 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:41.634324 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:41.634806 (+ 482us) client_negotiation.cc:769] Sending connection context
0928 02:17:41.635149 (+ 343us) client_negotiation.cc:240] Negotiation successful
0928 02:17:41.635749 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":252,"threads_started":1}
I20260928 02:17:41.639066 7288 heartbeater.cc:347] Connected to a master server at 127.4.140.254:32827
I20260928 02:17:41.639427 7288 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:41.640290 7288 heartbeater.cc:511] Master 127.4.140.254:32827 requested a full tablet report, sending...
I20260928 02:17:41.642709 7099 ts_manager.cc:194] Registered new tserver with Master: 34b26fef40064ab2875bf65ce8bfdb04 (127.4.140.193:45301)
WARNING: no policy specified for kudu/127.4.140.194@KRBTEST.COM; defaulting to no policy
I20260928 02:17:41.644332 7099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:54181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
I20260928 02:17:41.670881 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:32827
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:41.996091 7294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:41.996418 7294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:41.996588 7294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:42.003939 7294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:42.004117 7294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:42.004289 7294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:17:42.014170 7294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:32827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:42.015455 7294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:42.016821 7294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:42.030128 7302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:42.030246 7299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:42.030293 7300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:42.031050 7294 server_base.cc:1034] running on GCE node
I20260928 02:17:42.031654 7294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:42.032676 7294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:42.033898 7294 hybrid_clock.cc:648] HybridClock initialized: now 1790561862033839 us; error 68 us; skew 500 ppm
I20260928 02:17:42.034216 7294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:42 dist-test-slave-h37r krb5kdc[7054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561862, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:42.037576 7294 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:17:42.039106 7294 webserver.cc:460] Webserver started at http://127.4.140.194:41813/ using document root <none> and password file <none>
I20260928 02:17:42.039806 7294 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:42.039983 7294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:42.040294 7294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:42.043237 7294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "ace1d6e661894a5796d9312bb53a9b80"
format_stamp: "Formatted at 2026-09-28 02:17:42 on dist-test-slave-h37r"
server_key: "cb49b6a4d43e9277337a300a229d11a3"
server_key_iv: "258e6ca1eea3f7b4ee1fc033664b5672"
server_key_version: "encryptionkey@0"
I20260928 02:17:42.044162 7294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "ace1d6e661894a5796d9312bb53a9b80"
format_stamp: "Formatted at 2026-09-28 02:17:42 on dist-test-slave-h37r"
server_key: "cb49b6a4d43e9277337a300a229d11a3"
server_key_iv: "258e6ca1eea3f7b4ee1fc033664b5672"
server_key_version: "encryptionkey@0"
I20260928 02:17:42.049363 7294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 02:17:42.052932 7309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:42.054587 7294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:17:42.054754 7294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "ace1d6e661894a5796d9312bb53a9b80"
format_stamp: "Formatted at 2026-09-28 02:17:42 on dist-test-slave-h37r"
server_key: "cb49b6a4d43e9277337a300a229d11a3"
server_key_iv: "258e6ca1eea3f7b4ee1fc033664b5672"
server_key_version: "encryptionkey@0"
I20260928 02:17:42.054981 7294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:42.099187 7294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:42.103600 7294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:42.103853 7294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:42.105130 7294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:42.106575 7294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:42.106647 7294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:42.106746 7294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:42.106793 7294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:42.140498 7294 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:38053
I20260928 02:17:42.140615 7421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:38053 every 8 connection(s)
I20260928 02:17:42.141893 7294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:17:42.150243 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 7294
I20260928 02:17:42.150468 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:17:42.150995 4659 external_mini_cluster.cc:1428] Setting key e1639c8efe14b85d19501a2008b73b89
Sep 28 02:17:42 dist-test-slave-h37r krb5kdc[7054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561862, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:42.165066 7424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:42.144739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:51967 (local address 127.4.140.254:32827)
0928 02:17:42.145250 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:42.145254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:42.146371 (+ 1117us) server_negotiation.cc:407] Connection header received
0928 02:17:42.148403 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:42.148410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:42.148507 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:42.148645 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:42.150558 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:42.151394 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:42.153533 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:42.154039 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:42.154138 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:42.158898 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:42.158904 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:42.158916 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:42.158949 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:42.161842 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:42.162669 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:42.162674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:42.162680 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:42.162804 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:42.163248 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:42.163260 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:42.163263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:42.163510 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:42.163661 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:42.164447 (+ 786us) server_negotiation.cc:299] Negotiation successful
0928 02:17:42.164656 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":276,"threads_started":1}
I20260928 02:17:42.166091 7425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:42.145391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32827 (local address 127.4.140.194:51967)
0928 02:17:42.146170 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:42.146217 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:42.148108 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:42.148909 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:42.148918 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:42.149469 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:42.150308 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:42.150338 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:42.151554 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:42.151561 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:42.153354 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:42.153372 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:42.154348 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:42.154359 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:42.154787 (+ 428us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:42.155584 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:42.155623 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:42.158545 (+ 2922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:42.161997 (+ 3452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:42.162006 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:42.162026 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:42.162542 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:42.162916 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:42.162921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:42.162925 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:42.163127 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:42.163639 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:42.163647 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:42.164066 (+ 419us) client_negotiation.cc:769] Sending connection context
0928 02:17:42.164344 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 02:17:42.164901 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":301,"threads_started":1}
I20260928 02:17:42.167796 7422 heartbeater.cc:347] Connected to a master server at 127.4.140.254:32827
I20260928 02:17:42.168150 7422 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:42.168886 7422 heartbeater.cc:511] Master 127.4.140.254:32827 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.140.195@KRBTEST.COM; defaulting to no policy
I20260928 02:17:42.170421 7099 ts_manager.cc:194] Registered new tserver with Master: ace1d6e661894a5796d9312bb53a9b80 (127.4.140.194:38053)
I20260928 02:17:42.171232 7099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:51967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
I20260928 02:17:42.196786 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:32827
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:42.517776 7428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:42.518059 7428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:42.518151 7428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:42.525622 7428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:42.525727 7428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:42.525835 7428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:17:42.535590 7428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:32827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:42.536919 7428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:42.538210 7428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:42.551545 7434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:42.551433 7433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:42.551471 7436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:42.552364 7428 server_base.cc:1034] running on GCE node
I20260928 02:17:42.552923 7428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:42.553882 7428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:42.555085 7428 hybrid_clock.cc:648] HybridClock initialized: now 1790561862555029 us; error 59 us; skew 500 ppm
I20260928 02:17:42.555392 7428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:42 dist-test-slave-h37r krb5kdc[7054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561862, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:42.558358 7428 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:17:42.560221 7428 webserver.cc:460] Webserver started at http://127.4.140.195:38479/ using document root <none> and password file <none>
I20260928 02:17:42.561091 7428 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:42.561218 7428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:42.561523 7428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:42.564529 7428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "b39862e379574dd1905823a43bf147d8"
format_stamp: "Formatted at 2026-09-28 02:17:42 on dist-test-slave-h37r"
server_key: "2de444dc3200ba31e3eaeb66bd0b52bb"
server_key_iv: "f2c0d5f4ec4cf5e7a78aaeebe9dc79d7"
server_key_version: "encryptionkey@0"
I20260928 02:17:42.565245 7428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "b39862e379574dd1905823a43bf147d8"
format_stamp: "Formatted at 2026-09-28 02:17:42 on dist-test-slave-h37r"
server_key: "2de444dc3200ba31e3eaeb66bd0b52bb"
server_key_iv: "f2c0d5f4ec4cf5e7a78aaeebe9dc79d7"
server_key_version: "encryptionkey@0"
I20260928 02:17:42.570190 7428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 02:17:42.573616 7443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:42.574923 7428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:17:42.575110 7428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "b39862e379574dd1905823a43bf147d8"
format_stamp: "Formatted at 2026-09-28 02:17:42 on dist-test-slave-h37r"
server_key: "2de444dc3200ba31e3eaeb66bd0b52bb"
server_key_iv: "f2c0d5f4ec4cf5e7a78aaeebe9dc79d7"
server_key_version: "encryptionkey@0"
I20260928 02:17:42.575291 7428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:42.591640 7428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:42.595739 7428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:42.596057 7428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:42.597159 7428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:42.598551 7428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:42.598623 7428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:42.598778 7428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:42.598836 7428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:42.631637 7428 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:42593
I20260928 02:17:42.631681 7555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:42593 every 8 connection(s)
I20260928 02:17:42.632911 7428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:17:42.635061 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 7428
I20260928 02:17:42.635249 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:17:42.635778 4659 external_mini_cluster.cc:1428] Setting key 07ce6ef6182a901bc9c0c14c97217891
Sep 28 02:17:42 dist-test-slave-h37r krb5kdc[7054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561862, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:42.647464 7288 heartbeater.cc:503] Master 127.4.140.254:32827 was elected leader, sending a full tablet report...
I20260928 02:17:42.653502 7424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:42.635401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:42975 (local address 127.4.140.254:32827)
0928 02:17:42.635619 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:42.635626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:42.636867 (+ 1241us) server_negotiation.cc:407] Connection header received
0928 02:17:42.638701 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:42.638706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:42.638809 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:42.638921 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:42.640485 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:42.641214 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:42.642770 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:42.643225 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:42.643375 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:42.647000 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:42.647005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:42.647009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:42.647049 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:42.650344 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:42.651150 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:42.651155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:42.651159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:42.651287 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:42.651721 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:42.651726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:42.651729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:42.652090 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:42.652245 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:42.652942 (+ 697us) server_negotiation.cc:299] Negotiation successful
0928 02:17:42.653163 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:17:42.654536 7558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:42.635892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32827 (local address 127.4.140.195:42975)
0928 02:17:42.636724 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:42.636768 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:42.638437 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:42.639168 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:42.639176 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:42.639686 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:42.640324 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:42.640342 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:42.641379 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:42.641383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:42.642621 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:42.642633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:42.643476 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:42.643483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:42.643791 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:42.644546 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:42.644568 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:42.646798 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:42.650503 (+ 3705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:42.650510 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:42.650548 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:42.651024 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:42.651401 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:42.651406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:42.651416 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:42.651603 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:42.652214 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:42.652222 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:42.652588 (+ 366us) client_negotiation.cc:769] Sending connection context
0928 02:17:42.652851 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 02:17:42.653382 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":326,"threads_started":1}
I20260928 02:17:42.655982 7556 heartbeater.cc:347] Connected to a master server at 127.4.140.254:32827
I20260928 02:17:42.656323 7556 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:42.657058 7556 heartbeater.cc:511] Master 127.4.140.254:32827 requested a full tablet report, sending...
I20260928 02:17:42.658631 7099 ts_manager.cc:194] Registered new tserver with Master: b39862e379574dd1905823a43bf147d8 (127.4.140.195:42593)
I20260928 02:17:42.659595 7099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:42975
I20260928 02:17:42.663043 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:17:42.693122 7424 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:17:42.690357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34848 (local address 127.4.140.254:32827)
0928 02:17:42.690558 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:42.690567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:42.691045 (+ 478us) server_negotiation.cc:407] Connection header received
0928 02:17:42.691284 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:42.691288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:42.691354 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:42.691461 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:42.692796 (+ 1335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34848: BlockingRecv error: recv got EOF from 127.0.0.1:34848 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:17:42.696110 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 7160
W20260928 02:17:42.699026 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 7160: Not authorized: Client connection negotiation failed: client connection to 127.4.140.193:45301: server requires authentication, but client does not have Kerberos credentials available
I20260928 02:17:42.699211 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 7160
I20260928 02:17:42.714488 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 7294
W20260928 02:17:42.717166 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 7294: Not authorized: Client connection negotiation failed: client connection to 127.4.140.194:38053: server requires authentication, but client does not have Kerberos credentials available
I20260928 02:17:42.717314 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 7294
I20260928 02:17:42.732667 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 7428
W20260928 02:17:42.735440 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 7428: Not authorized: Client connection negotiation failed: client connection to 127.4.140.195:42593: server requires authentication, but client does not have Kerberos credentials available
I20260928 02:17:42.735625 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 7428
I20260928 02:17:42.751055 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 7067
I20260928 02:17:42.752427 7129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:42.856153 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 7067
2026-09-28T02:17:42Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3571 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:17:42 dist-test-slave-h37r krb5kdc[7573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:17:42 dist-test-slave-h37r krb5kdc[7573](info): set up 2 sockets
Sep 28 02:17:42 dist-test-slave-h37r krb5kdc[7573](info): commencing operation
krb5kdc: starting...
W20260928 02:17:43.956400 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:17:44 dist-test-slave-h37r krb5kdc[7573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:17:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:17:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:17:44.151418 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:32813
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:32813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:44.470870 7586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:44.471174 7586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:44.471352 7586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:44.478595 7586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:17:44.478699 7586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:44.478816 7586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:44.478920 7586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:17:44.478993 7586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:17:44.488749 7586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:32813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:32813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:44.490077 7586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:44.491400 7586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:44.503578 7594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:44.503857 7592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:44.503861 7591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:44.505376 7586 server_base.cc:1034] running on GCE node
I20260928 02:17:44.506160 7586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:44.507598 7586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:44.508793 7586 hybrid_clock.cc:648] HybridClock initialized: now 1790561864508722 us; error 60 us; skew 500 ppm
I20260928 02:17:44.509152 7586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:44 dist-test-slave-h37r krb5kdc[7573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561864, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:44.522219 7586 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:17:44.524011 7586 webserver.cc:460] Webserver started at http://127.4.140.254:43489/ using document root <none> and password file <none>
I20260928 02:17:44.524768 7586 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:44.524922 7586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:44.525239 7586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:44.528329 7586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "a8a5d7aec9934aa187bd72404479e30a"
format_stamp: "Formatted at 2026-09-28 02:17:44 on dist-test-slave-h37r"
server_key: "58ecc7db78353254d81d130b073465ea"
server_key_iv: "000ee995cec1a9ccba2b48a73d3a45f2"
server_key_version: "encryptionkey@0"
I20260928 02:17:44.529114 7586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "a8a5d7aec9934aa187bd72404479e30a"
format_stamp: "Formatted at 2026-09-28 02:17:44 on dist-test-slave-h37r"
server_key: "58ecc7db78353254d81d130b073465ea"
server_key_iv: "000ee995cec1a9ccba2b48a73d3a45f2"
server_key_version: "encryptionkey@0"
I20260928 02:17:44.534185 7586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:17:44.537616 7601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:44.538960 7586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:17:44.539139 7586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "a8a5d7aec9934aa187bd72404479e30a"
format_stamp: "Formatted at 2026-09-28 02:17:44 on dist-test-slave-h37r"
server_key: "58ecc7db78353254d81d130b073465ea"
server_key_iv: "000ee995cec1a9ccba2b48a73d3a45f2"
server_key_version: "encryptionkey@0"
I20260928 02:17:44.539330 7586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:44.565289 7586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:44.569775 7586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:44.570075 7586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:44.588985 7586 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:32813
I20260928 02:17:44.589018 7652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:32813 every 8 connection(s)
I20260928 02:17:44.590363 7586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:17:44.594197 7653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:44.600406 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 7586
I20260928 02:17:44.600634 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:17:44.601434 4659 external_mini_cluster.cc:1428] Setting key 72c6edf1521f187ef23739212d1e4fc0
I20260928 02:17:44.605554 7653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a: Bootstrap starting.
I20260928 02:17:44.609371 7653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:44.610550 7653 log.cc:824] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:44.613819 7653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a: No bootstrap required, opened a new log
Sep 28 02:17:44 dist-test-slave-h37r krb5kdc[7573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:44.625192 7653 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a5d7aec9934aa187bd72404479e30a" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32813 } }
I20260928 02:17:44.625680 7653 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:44.625806 7653 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a5d7aec9934aa187bd72404479e30a, State: Initialized, Role: FOLLOWER
I20260928 02:17:44.626562 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:44.603354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34992 (local address 127.4.140.254:32813)
0928 02:17:44.604201 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:44.604214 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:44.604254 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:17:44.605992 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:44.606021 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:44.606416 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:44.606749 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:44.608649 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:44.610025 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:44.611698 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:44.612469 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:44.612638 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:44.617665 (+ 5027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:44.617676 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:44.617703 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:44.617758 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:44.621842 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:44.622542 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:44.622551 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:44.622560 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:44.622782 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:44.623135 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:44.623140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:44.623143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:44.623934 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:44.624221 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:44.624594 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 02:17:44.625061 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":387,"threads_started":1}
I20260928 02:17:44.626530 7653 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a5d7aec9934aa187bd72404479e30a" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32813 } }
I20260928 02:17:44.626801 7653 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:17:44.626942 7653 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:17:44.627143 7653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:44.629818 7653 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a5d7aec9934aa187bd72404479e30a" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32813 } }
I20260928 02:17:44.630330 7653 leader_election.cc:302] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8a5d7aec9934aa187bd72404479e30a; no voters:
I20260928 02:17:44.630683 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:17:44.630995 7653 leader_election.cc:288] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:17:44.631127 7658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:44.632972 7653 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:17:44.640532 7658 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a [term 1 LEADER]: Becoming Leader. State: Replica: a8a5d7aec9934aa187bd72404479e30a, State: Running, Role: LEADER
I20260928 02:17:44.641323 7658 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a5d7aec9934aa187bd72404479e30a" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32813 } }
I20260928 02:17:44.644318 7660 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8a5d7aec9934aa187bd72404479e30a. Latest consensus state: current_term: 1 leader_uuid: "a8a5d7aec9934aa187bd72404479e30a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a5d7aec9934aa187bd72404479e30a" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32813 } } }
I20260928 02:17:44.644698 7660 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:44.645284 7659 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8a5d7aec9934aa187bd72404479e30a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a5d7aec9934aa187bd72404479e30a" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32813 } } }
I20260928 02:17:44.645596 7659 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:44.664610 7672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:17:44.671344 7672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:17:44.681612 7672 catalog_manager.cc:1347] Generated new cluster ID: d375a850a28647a49fe137a125c03e03
I20260928 02:17:44.681810 7672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:17:44.694129 7672 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:17:44.696125 7672 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:17:44.703923 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:44.693519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35004 (local address 127.4.140.254:32813)
0928 02:17:44.693738 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:44.693747 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:44.696531 (+ 2784us) server_negotiation.cc:407] Connection header received
0928 02:17:44.696788 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:44.696793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:44.696912 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:44.697036 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:44.698442 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:44.699178 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:44.700580 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:44.701117 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:44.701289 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:44.702420 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:44.702433 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:44.702436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:17:44.703068 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:44.703125 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:17:44.703151 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:44.703275 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:44.703413 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 02:17:44.703504 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:17:44.710319 7672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8a5d7aec9934aa187bd72404479e30a: Generated new TSK 0
I20260928 02:17:44.711381 7672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:17:44.721480 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 7586
I20260928 02:17:44.722620 7648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:44.813048 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 7586
2026-09-28T02:17:44Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1951 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:17:44 dist-test-slave-h37r krb5kdc[7688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:17:44 dist-test-slave-h37r krb5kdc[7688](info): set up 2 sockets
Sep 28 02:17:44 dist-test-slave-h37r krb5kdc[7688](info): commencing operation
krb5kdc: starting...
W20260928 02:17:45.892416 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:17:46 dist-test-slave-h37r krb5kdc[7688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:17:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:17:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:17:46.089871 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:43471
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:42955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:43471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:46.408881 7701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:46.409169 7701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:46.409257 7701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:46.416326 7701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:17:46.416437 7701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:46.416482 7701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:46.416538 7701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:17:46.416574 7701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:17:46.426435 7701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:42955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:43471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:43471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:46.427843 7701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:46.429278 7701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:46.441538 7706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:46.441576 7709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:46.441637 7707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:46.442260 7701 server_base.cc:1034] running on GCE node
I20260928 02:17:46.443032 7701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:46.444669 7701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:46.445894 7701 hybrid_clock.cc:648] HybridClock initialized: now 1790561866445849 us; error 35 us; skew 500 ppm
I20260928 02:17:46.446280 7701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:46 dist-test-slave-h37r krb5kdc[7688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561866, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:46.449622 7701 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:17:46.451188 7701 webserver.cc:460] Webserver started at http://127.4.140.254:38261/ using document root <none> and password file <none>
I20260928 02:17:46.451999 7701 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:46.452153 7701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:46.452471 7701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:46.455639 7701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "869a033c0190465cb1bb3e59e57b173b"
format_stamp: "Formatted at 2026-09-28 02:17:46 on dist-test-slave-h37r"
server_key: "1ee97dcdc9b5b8906c4ab7066152bada"
server_key_iv: "f5ecf036fd2d5781cf791c3ed03bbc51"
server_key_version: "encryptionkey@0"
I20260928 02:17:46.456489 7701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "869a033c0190465cb1bb3e59e57b173b"
format_stamp: "Formatted at 2026-09-28 02:17:46 on dist-test-slave-h37r"
server_key: "1ee97dcdc9b5b8906c4ab7066152bada"
server_key_iv: "f5ecf036fd2d5781cf791c3ed03bbc51"
server_key_version: "encryptionkey@0"
I20260928 02:17:46.462122 7701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:17:46.465821 7716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:46.467229 7701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:17:46.467427 7701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "869a033c0190465cb1bb3e59e57b173b"
format_stamp: "Formatted at 2026-09-28 02:17:46 on dist-test-slave-h37r"
server_key: "1ee97dcdc9b5b8906c4ab7066152bada"
server_key_iv: "f5ecf036fd2d5781cf791c3ed03bbc51"
server_key_version: "encryptionkey@0"
I20260928 02:17:46.467645 7701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:46.485116 7701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:46.489373 7701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:46.489674 7701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:46.508719 7701 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:43471
I20260928 02:17:46.508766 7767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:43471 every 8 connection(s)
I20260928 02:17:46.510188 7701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:17:46.514000 7768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:46.519132 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 7701
I20260928 02:17:46.519372 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:17:46.520129 4659 external_mini_cluster.cc:1428] Setting key 34c357e7e39f92ba46609d2c4b7890f0
I20260928 02:17:46.525620 7768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b: Bootstrap starting.
I20260928 02:17:46.528765 7768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:46.529799 7768 log.cc:824] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:46.533372 7768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b: No bootstrap required, opened a new log
Sep 28 02:17:46 dist-test-slave-h37r krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:46.541599 7768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869a033c0190465cb1bb3e59e57b173b" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43471 } }
I20260928 02:17:46.542057 7768 raft_consensus.cc:374] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:46.542181 7768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869a033c0190465cb1bb3e59e57b173b, State: Initialized, Role: FOLLOWER
I20260928 02:17:46.542795 7768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869a033c0190465cb1bb3e59e57b173b" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43471 } }
I20260928 02:17:46.543040 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:46.522265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40094 (local address 127.4.140.254:43471)
0928 02:17:46.523073 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:46.523085 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:46.523131 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:17:46.524864 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:46.524887 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:46.525294 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:46.525693 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:46.527076 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:46.528431 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:46.529776 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:46.530777 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:46.531000 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:46.535409 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:46.535419 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:46.535435 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:46.535475 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:46.538543 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:46.539248 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:46.539255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:46.539265 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:46.539432 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:46.539839 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:46.539844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:46.539849 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:46.540467 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:46.540813 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:46.541276 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 02:17:46.541749 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":359,"threads_started":1}
I20260928 02:17:46.543054 7768 raft_consensus.cc:388] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:17:46.543268 7768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:17:46.543534 7768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:46.545974 7768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869a033c0190465cb1bb3e59e57b173b" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43471 } }
I20260928 02:17:46.546482 7768 leader_election.cc:302] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 869a033c0190465cb1bb3e59e57b173b; no voters:
I20260928 02:17:46.547040 7768 leader_election.cc:288] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:17:46.547178 7773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:46.547811 7773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b [term 1 LEADER]: Becoming Leader. State: Replica: 869a033c0190465cb1bb3e59e57b173b, State: Running, Role: LEADER
I20260928 02:17:46.548445 7773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869a033c0190465cb1bb3e59e57b173b" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43471 } }
I20260928 02:17:46.549134 7768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:17:46.551492 7775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 869a033c0190465cb1bb3e59e57b173b. Latest consensus state: current_term: 1 leader_uuid: "869a033c0190465cb1bb3e59e57b173b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869a033c0190465cb1bb3e59e57b173b" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43471 } } }
I20260928 02:17:46.551874 7775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:46.551605 7774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "869a033c0190465cb1bb3e59e57b173b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869a033c0190465cb1bb3e59e57b173b" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43471 } } }
I20260928 02:17:46.552109 7774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:46.556581 7780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:17:46.561673 7780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:17:46.570317 7780 catalog_manager.cc:1347] Generated new cluster ID: 85bcb90f3050469ab6b1abf99678e23c
I20260928 02:17:46.570448 7780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:17:46.582436 7780 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:17:46.583710 7780 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:17:46.593974 7780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 869a033c0190465cb1bb3e59e57b173b: Generated new TSK 0
I20260928 02:17:46.594995 7780 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
I20260928 02:17:46.648154 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43471
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:42955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:46.984015 7794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:46.984364 7794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:46.984459 7794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:46.991946 7794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:46.992087 7794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:46.992213 7794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:17:47.002275 7794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:42955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:47.003597 7794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:47.004917 7794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:47.017976 7799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:47.018177 7800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:47.018190 7802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:47.019008 7794 server_base.cc:1034] running on GCE node
I20260928 02:17:47.019630 7794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:47.020603 7794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:47.021819 7794 hybrid_clock.cc:648] HybridClock initialized: now 1790561867021768 us; error 59 us; skew 500 ppm
I20260928 02:17:47.022131 7794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:47 dist-test-slave-h37r krb5kdc[7688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561867, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:47.025367 7794 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:17:47.026911 7794 webserver.cc:460] Webserver started at http://127.4.140.193:34453/ using document root <none> and password file <none>
I20260928 02:17:47.027583 7794 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:47.027719 7794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:47.028067 7794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:47.030858 7794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "8546ad42883c45b8a4713e1822f88088"
format_stamp: "Formatted at 2026-09-28 02:17:47 on dist-test-slave-h37r"
server_key: "c15f7647cbe44df22437957f50dd4a68"
server_key_iv: "4d58a0cdd58c31b1f3cce5182cc496c7"
server_key_version: "encryptionkey@0"
I20260928 02:17:47.031644 7794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "8546ad42883c45b8a4713e1822f88088"
format_stamp: "Formatted at 2026-09-28 02:17:47 on dist-test-slave-h37r"
server_key: "c15f7647cbe44df22437957f50dd4a68"
server_key_iv: "4d58a0cdd58c31b1f3cce5182cc496c7"
server_key_version: "encryptionkey@0"
I20260928 02:17:47.037243 7794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 02:17:47.040623 7809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:47.041941 7794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:17:47.042115 7794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "8546ad42883c45b8a4713e1822f88088"
format_stamp: "Formatted at 2026-09-28 02:17:47 on dist-test-slave-h37r"
server_key: "c15f7647cbe44df22437957f50dd4a68"
server_key_iv: "4d58a0cdd58c31b1f3cce5182cc496c7"
server_key_version: "encryptionkey@0"
I20260928 02:17:47.042299 7794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:47.059152 7794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:47.063231 7794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:47.063509 7794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:47.064463 7794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:47.065824 7794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:47.065922 7794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:47.066067 7794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:47.066138 7794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:47.099758 7794 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:34405
I20260928 02:17:47.099807 7921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:34405 every 8 connection(s)
I20260928 02:17:47.101194 7794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:17:47.107715 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 7794
I20260928 02:17:47.107978 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:17:47.108769 4659 external_mini_cluster.cc:1428] Setting key eb755c6de1ce67d80e1dbf557af76042
Sep 28 02:17:47 dist-test-slave-h37r krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561867, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:47.123060 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:47.103710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:39607 (local address 127.4.140.254:43471)
0928 02:17:47.104183 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:47.104188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:47.105177 (+ 989us) server_negotiation.cc:407] Connection header received
0928 02:17:47.107083 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:47.107088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:47.107162 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:47.107253 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:47.109436 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:47.110227 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:47.112640 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:47.113031 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:47.113130 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:47.117391 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:47.117397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:47.117400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:47.117430 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:47.120030 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:47.120744 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:47.120749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:47.120755 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:47.120876 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:47.121316 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:47.121321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:47.121324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:47.121587 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:47.121746 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:47.122456 (+ 710us) server_negotiation.cc:299] Negotiation successful
0928 02:17:47.122682 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":256,"threads_started":1}
I20260928 02:17:47.124241 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:47.104321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43471 (local address 127.4.140.193:39607)
0928 02:17:47.105023 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:47.105060 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:47.106740 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:47.107566 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:47.107580 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:47.108383 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:47.109237 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:47.109262 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:47.110444 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:47.110449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:47.112478 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:47.112496 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:47.113258 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:47.113267 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:47.113597 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:47.114351 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:47.114409 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:47.117139 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:47.120162 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:47.120171 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:47.120190 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:47.120611 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:47.120987 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:47.120992 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:47.120996 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:47.121190 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:47.121719 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:47.121727 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:47.122081 (+ 354us) client_negotiation.cc:769] Sending connection context
0928 02:17:47.122342 (+ 261us) client_negotiation.cc:240] Negotiation successful
0928 02:17:47.122906 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":278,"threads_started":1}
I20260928 02:17:47.125948 7922 heartbeater.cc:347] Connected to a master server at 127.4.140.254:43471
I20260928 02:17:47.126263 7922 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.140.194@KRBTEST.COM; defaulting to no policy
I20260928 02:17:47.127022 7922 heartbeater.cc:511] Master 127.4.140.254:43471 requested a full tablet report, sending...
I20260928 02:17:47.129242 7733 ts_manager.cc:194] Registered new tserver with Master: 8546ad42883c45b8a4713e1822f88088 (127.4.140.193:34405)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.194@KRBTEST.COM" created.
I20260928 02:17:47.130731 7733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:39607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
I20260928 02:17:47.155850 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43471
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:42955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:47.477349 7928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:47.477638 7928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:47.477731 7928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:47.484829 7928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:47.484941 7928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:47.485051 7928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:17:47.494939 7928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:42955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:47.496304 7928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:47.497556 7928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:47.511031 7933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:47.511045 7934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:47.511709 7936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:47.512755 7928 server_base.cc:1034] running on GCE node
I20260928 02:17:47.513445 7928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:47.514566 7928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:47.515853 7928 hybrid_clock.cc:648] HybridClock initialized: now 1790561867515792 us; error 49 us; skew 500 ppm
I20260928 02:17:47.516300 7928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:47 dist-test-slave-h37r krb5kdc[7688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561867, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:47.520619 7928 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:17:47.522353 7928 webserver.cc:460] Webserver started at http://127.4.140.194:36979/ using document root <none> and password file <none>
I20260928 02:17:47.523100 7928 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:47.523252 7928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:47.523581 7928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:47.526703 7928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "6805e4b7b7ea40f7acffc7ebc9d28885"
format_stamp: "Formatted at 2026-09-28 02:17:47 on dist-test-slave-h37r"
server_key: "47a262303827bf91d27fc34551ab457d"
server_key_iv: "ecd8ea225a5c39a49b67b4deb7df1ca5"
server_key_version: "encryptionkey@0"
I20260928 02:17:47.527565 7928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "6805e4b7b7ea40f7acffc7ebc9d28885"
format_stamp: "Formatted at 2026-09-28 02:17:47 on dist-test-slave-h37r"
server_key: "47a262303827bf91d27fc34551ab457d"
server_key_iv: "ecd8ea225a5c39a49b67b4deb7df1ca5"
server_key_version: "encryptionkey@0"
I20260928 02:17:47.533048 7928 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 02:17:47.536873 7943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:47.538302 7928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:17:47.538492 7928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "6805e4b7b7ea40f7acffc7ebc9d28885"
format_stamp: "Formatted at 2026-09-28 02:17:47 on dist-test-slave-h37r"
server_key: "47a262303827bf91d27fc34551ab457d"
server_key_iv: "ecd8ea225a5c39a49b67b4deb7df1ca5"
server_key_version: "encryptionkey@0"
I20260928 02:17:47.538700 7928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:47.561760 7928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:47.566054 7928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:47.566354 7928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:47.567345 7928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:47.568786 7928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:47.568893 7928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:47.569048 7928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:47.569121 7928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:47.602267 7928 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:43121
I20260928 02:17:47.602308 8055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:43121 every 8 connection(s)
I20260928 02:17:47.603641 7928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:17:47.604801 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 7928
I20260928 02:17:47.604998 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:17:47.605580 4659 external_mini_cluster.cc:1428] Setting key 6d88481a120d95bbf855e96f7b816f57
WARNING: no policy specified for kudu/127.4.140.195@KRBTEST.COM; defaulting to no policy
Sep 28 02:17:47 dist-test-slave-h37r krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561867, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.195@KRBTEST.COM" created.
I20260928 02:17:47.637698 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:47.610728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:37703 (local address 127.4.140.254:43471)
0928 02:17:47.610967 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:47.610976 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:47.614973 (+ 3997us) server_negotiation.cc:407] Connection header received
0928 02:17:47.616969 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:47.616975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:47.617062 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:47.617185 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:47.619427 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:47.620691 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:47.623775 (+ 3084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:47.624611 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:47.624751 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:47.630128 (+ 5377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:47.630133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:47.630136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:47.630170 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:47.633767 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:47.634795 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:47.634803 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:47.634809 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:47.634973 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:47.635577 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:47.635584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:47.635590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:47.636010 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:47.636161 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:47.637080 (+ 919us) server_negotiation.cc:299] Negotiation successful
0928 02:17:47.637341 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:17:47.639057 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:47.608975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43471 (local address 127.4.140.194:37703)
0928 02:17:47.614768 (+ 5793us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:47.614823 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:47.616678 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:47.617459 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:47.617473 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:47.618300 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:47.619194 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:47.619230 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:47.620891 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:47.620900 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:47.623585 (+ 2685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:47.623605 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:47.625083 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:47.625092 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:47.625469 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:47.626411 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:47.626445 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:47.629868 (+ 3423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:47.633933 (+ 4065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:47.633947 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:47.633968 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:47.634659 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:47.635120 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:47.635128 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:47.635135 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:47.635443 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:47.636183 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:47.636198 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:47.636666 (+ 468us) client_negotiation.cc:769] Sending connection context
0928 02:17:47.636992 (+ 326us) client_negotiation.cc:240] Negotiation successful
0928 02:17:47.637566 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5485,"thread_start_us":5225,"threads_started":1}
I20260928 02:17:47.640563 8056 heartbeater.cc:347] Connected to a master server at 127.4.140.254:43471
I20260928 02:17:47.640882 8056 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:47.641626 8056 heartbeater.cc:511] Master 127.4.140.254:43471 requested a full tablet report, sending...
I20260928 02:17:47.643256 7733 ts_manager.cc:194] Registered new tserver with Master: 6805e4b7b7ea40f7acffc7ebc9d28885 (127.4.140.194:43121)
I20260928 02:17:47.644300 7733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:37703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
I20260928 02:17:47.660367 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43471
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:42955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:48.009912 8061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:48.010259 8061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:48.010362 8061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:48.017549 8061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:48.017666 8061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:48.017782 8061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:17:48.027616 8061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:42955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:48.028970 8061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:48.030268 8061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:48.043659 8066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:48.043686 8067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:48.043740 8069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:48.044492 8061 server_base.cc:1034] running on GCE node
I20260928 02:17:48.045116 8061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:48.046201 8061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:48.047403 8061 hybrid_clock.cc:648] HybridClock initialized: now 1790561868047356 us; error 88 us; skew 500 ppm
I20260928 02:17:48.047676 8061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:48 dist-test-slave-h37r krb5kdc[7688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561868, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:48.051144 8061 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:17:48.052754 8061 webserver.cc:460] Webserver started at http://127.4.140.195:43533/ using document root <none> and password file <none>
I20260928 02:17:48.053484 8061 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:48.053586 8061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:48.053836 8061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:48.057051 8061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "6e8d483eb0c04eeb9db3e5cb9fae021d"
format_stamp: "Formatted at 2026-09-28 02:17:48 on dist-test-slave-h37r"
server_key: "8de2be8228d83bcf79f7902f15c0f6e2"
server_key_iv: "d7b76c26872d03cc5c28a919462d9ba4"
server_key_version: "encryptionkey@0"
I20260928 02:17:48.057791 8061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "6e8d483eb0c04eeb9db3e5cb9fae021d"
format_stamp: "Formatted at 2026-09-28 02:17:48 on dist-test-slave-h37r"
server_key: "8de2be8228d83bcf79f7902f15c0f6e2"
server_key_iv: "d7b76c26872d03cc5c28a919462d9ba4"
server_key_version: "encryptionkey@0"
I20260928 02:17:48.062933 8061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:17:48.066336 8076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:48.067628 8061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:17:48.067752 8061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "6e8d483eb0c04eeb9db3e5cb9fae021d"
format_stamp: "Formatted at 2026-09-28 02:17:48 on dist-test-slave-h37r"
server_key: "8de2be8228d83bcf79f7902f15c0f6e2"
server_key_iv: "d7b76c26872d03cc5c28a919462d9ba4"
server_key_version: "encryptionkey@0"
I20260928 02:17:48.067978 8061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:48.087419 8061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:48.091506 8061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:48.091799 8061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:48.092865 8061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:48.094188 8061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:48.094259 8061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:48.094342 8061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:48.094388 8061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:48.128082 8061 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:35093
I20260928 02:17:48.128125 8188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:35093 every 8 connection(s)
I20260928 02:17:48.129422 8061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:17:48.133258 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 8061
I20260928 02:17:48.133452 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:17:48.133622 7922 heartbeater.cc:503] Master 127.4.140.254:43471 was elected leader, sending a full tablet report...
I20260928 02:17:48.133951 4659 external_mini_cluster.cc:1428] Setting key a7c894a802f211e553ddba053feadcc8
Sep 28 02:17:48 dist-test-slave-h37r krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561868, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:48.150542 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:48.132005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:51559 (local address 127.4.140.254:43471)
0928 02:17:48.132179 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:48.132184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:48.133536 (+ 1352us) server_negotiation.cc:407] Connection header received
0928 02:17:48.136066 (+ 2530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:48.136072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:48.136169 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:48.136267 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:48.138074 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:48.138801 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:48.140342 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:48.140771 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:48.140866 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:48.144407 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:48.144411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:48.144415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:48.144445 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:48.147120 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:48.147885 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:48.147889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:48.147928 (+ 39us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:48.148114 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:48.148633 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:48.148637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:48.148640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:48.149013 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:48.149175 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:48.150063 (+ 888us) server_negotiation.cc:299] Negotiation successful
0928 02:17:48.150223 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:17:48.151541 8191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:48.132575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43471 (local address 127.4.140.195:51559)
0928 02:17:48.133356 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:48.133424 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:48.135744 (+ 2320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:48.136617 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:48.136629 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:48.137342 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:48.137915 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:48.137932 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:48.138921 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:48.138924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:48.140204 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:48.140214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:48.140958 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:48.140962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:48.141150 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:48.141780 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:48.141805 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:48.144224 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:48.147291 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:48.147302 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:48.147323 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:48.147756 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:48.148233 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:48.148239 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:48.148242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:48.148504 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:48.149180 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:48.149205 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:48.149630 (+ 425us) client_negotiation.cc:769] Sending connection context
0928 02:17:48.149903 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 02:17:48.150397 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":328,"threads_started":1}
I20260928 02:17:48.152954 8189 heartbeater.cc:347] Connected to a master server at 127.4.140.254:43471
I20260928 02:17:48.153228 8189 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:48.153927 8189 heartbeater.cc:511] Master 127.4.140.254:43471 requested a full tablet report, sending...
I20260928 02:17:48.155352 7732 ts_manager.cc:194] Registered new tserver with Master: 6e8d483eb0c04eeb9db3e5cb9fae021d (127.4.140.195:35093)
I20260928 02:17:48.156347 7732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:51559
I20260928 02:17:48.161993 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:17:48 dist-test-slave-h37r krb5kdc[7688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561868, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 02:17:48 dist-test-slave-h37r krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561868, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:48.210038 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:48.196176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40102 (local address 127.4.140.254:43471)
0928 02:17:48.196436 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:48.196443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:48.196770 (+ 327us) server_negotiation.cc:407] Connection header received
0928 02:17:48.196916 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:48.196920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:48.196985 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:48.197072 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:48.198385 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:48.199123 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:48.200412 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:48.200819 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:48.200920 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:48.204715 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:48.204719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:48.204723 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:48.204759 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:48.207381 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:48.208048 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:48.208052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:48.208054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:48.208174 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:48.208548 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:48.208553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:48.208555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:48.208937 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:48.209149 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:48.209480 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 02:17:48.209672 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 02:17:48.210742 7732 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40102
I20260928 02:17:48.212534 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 7794
Sep 28 02:17:48 dist-test-slave-h37r krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561868, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:17:48.228552 8200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:48.213662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32798 (local address 127.4.140.193:34405)
0928 02:17:48.214317 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:48.214323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:48.214349 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:17:48.214403 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:48.214409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:48.214626 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:48.214747 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:48.215878 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:48.216957 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:48.218095 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:48.218668 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:48.218763 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:48.222737 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:48.222750 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:48.222766 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:48.222801 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:48.225752 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:48.226510 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:48.226517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:48.226519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:48.226647 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:48.226982 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:48.226985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:48.226987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:48.227334 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:48.227598 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:48.227842 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 02:17:48.228111 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":351,"threads_started":1}
W20260928 02:17:48.229947 7917 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:32798
W20260928 02:17:48.231168 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 7794: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:17:48.231369 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 7794
I20260928 02:17:48.247506 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 7928
Sep 28 02:17:48 dist-test-slave-h37r krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561868, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.140.194@KRBTEST.COM
I20260928 02:17:48.264381 8201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:48.248428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57364 (local address 127.4.140.194:43121)
0928 02:17:48.249005 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:48.249013 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:48.249041 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:17:48.249155 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:48.249165 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:48.249493 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:48.249638 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:48.250880 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:48.252024 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:48.253251 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:48.254037 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:48.254147 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:48.258483 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:48.258496 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:48.258512 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:48.258556 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:48.261828 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:48.262437 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:48.262445 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:48.262448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:48.262593 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:48.262904 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:48.262908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:48.262910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:48.263278 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:48.263531 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:48.263767 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 02:17:48.264064 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":308,"threads_started":1}
W20260928 02:17:48.265710 8051 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57364
W20260928 02:17:48.266853 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 7928: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:17:48.267033 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 7928
I20260928 02:17:48.282977 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 8061
Sep 28 02:17:48 dist-test-slave-h37r krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561868, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.140.195@KRBTEST.COM
I20260928 02:17:48.301106 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:48.284085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44068 (local address 127.4.140.195:35093)
0928 02:17:48.284719 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:48.284726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:48.284754 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:17:48.284815 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:48.284821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:48.285028 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:48.285211 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:48.286570 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:48.287605 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:48.288909 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:48.289482 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:48.289655 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:48.294073 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:48.294086 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:48.294102 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:48.294143 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:48.297918 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:48.298795 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:48.298808 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:48.298812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:48.299025 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:48.299476 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:48.299481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:48.299483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:48.300009 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:48.300277 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:48.300549 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 02:17:48.300774 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":344,"threads_started":1}
W20260928 02:17:48.302518 8184 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44068
W20260928 02:17:48.303805 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 8061: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:17:48.304039 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 8061
I20260928 02:17:48.320039 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 7701
I20260928 02:17:48.321354 7763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:48.423821 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 7701
2026-09-28T02:17:48Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3616 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:17:48 dist-test-slave-h37r krb5kdc[8206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:17:48 dist-test-slave-h37r krb5kdc[8206](info): set up 2 sockets
Sep 28 02:17:48 dist-test-slave-h37r krb5kdc[8206](info): commencing operation
krb5kdc: starting...
W20260928 02:17:49.508577 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:17:49 dist-test-slave-h37r krb5kdc[8206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:17:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:17:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:17:49.716894 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:35867
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:35867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:50.053373 8219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:50.053668 8219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:50.053853 8219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:50.061009 8219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:17:50.061152 8219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:50.061244 8219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:50.061349 8219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:17:50.061419 8219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:17:50.071269 8219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:35867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:35867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:50.072599 8219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:50.073923 8219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:50.085677 8227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:50.085707 8225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:50.085891 8224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:50.086704 8219 server_base.cc:1034] running on GCE node
I20260928 02:17:50.087435 8219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:50.088953 8219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:50.090181 8219 hybrid_clock.cc:648] HybridClock initialized: now 1790561870090130 us; error 37 us; skew 500 ppm
I20260928 02:17:50.090528 8219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:50 dist-test-slave-h37r krb5kdc[8206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561870, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:50.099433 8219 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:17:50.101030 8219 webserver.cc:460] Webserver started at http://127.4.140.254:42075/ using document root <none> and password file <none>
I20260928 02:17:50.101744 8219 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:50.101874 8219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:50.102210 8219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:50.105261 8219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "e8acd1af364f4ccab77157472aef21e0"
format_stamp: "Formatted at 2026-09-28 02:17:50 on dist-test-slave-h37r"
server_key: "1571b8d512badc0f9b6b8770667dd655"
server_key_iv: "bab1e51ed8745d439590af7211332b90"
server_key_version: "encryptionkey@0"
I20260928 02:17:50.106047 8219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "e8acd1af364f4ccab77157472aef21e0"
format_stamp: "Formatted at 2026-09-28 02:17:50 on dist-test-slave-h37r"
server_key: "1571b8d512badc0f9b6b8770667dd655"
server_key_iv: "bab1e51ed8745d439590af7211332b90"
server_key_version: "encryptionkey@0"
I20260928 02:17:50.111315 8219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260928 02:17:50.114857 8234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:50.116286 8219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 02:17:50.116474 8219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "e8acd1af364f4ccab77157472aef21e0"
format_stamp: "Formatted at 2026-09-28 02:17:50 on dist-test-slave-h37r"
server_key: "1571b8d512badc0f9b6b8770667dd655"
server_key_iv: "bab1e51ed8745d439590af7211332b90"
server_key_version: "encryptionkey@0"
I20260928 02:17:50.116670 8219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:50.153671 8219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:50.157747 8219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:50.158046 8219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:50.177009 8219 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:35867
I20260928 02:17:50.177052 8285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:35867 every 8 connection(s)
I20260928 02:17:50.178401 8219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:17:50.182112 8286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:50.187208 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 8219
I20260928 02:17:50.187402 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:17:50.188192 4659 external_mini_cluster.cc:1428] Setting key 3f5b92ff3890f625b141ad5a4c57fc7f
I20260928 02:17:50.193346 8286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0: Bootstrap starting.
I20260928 02:17:50.197178 8286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:50.198450 8286 log.cc:824] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:50.202414 8286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0: No bootstrap required, opened a new log
Sep 28 02:17:50 dist-test-slave-h37r krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:50.211669 8286 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8acd1af364f4ccab77157472aef21e0" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 35867 } }
I20260928 02:17:50.212198 8286 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:50.212363 8286 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8acd1af364f4ccab77157472aef21e0, State: Initialized, Role: FOLLOWER
I20260928 02:17:50.212980 8286 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8acd1af364f4ccab77157472aef21e0" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 35867 } }
I20260928 02:17:50.213289 8286 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:17:50.213450 8286 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:17:50.213663 8286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:50.213680 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:50.190222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33702 (local address 127.4.140.254:35867)
0928 02:17:50.191091 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:50.191106 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:50.191152 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:17:50.192895 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:50.192922 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:50.193305 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:50.193672 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:50.195515 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:50.196855 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:50.198699 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:50.199878 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:50.200072 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:50.205118 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:50.205129 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:50.205145 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:50.205189 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:50.208840 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:50.209750 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:50.209763 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:50.209773 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:50.209980 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:50.210372 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:50.210379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:50.210387 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:50.210991 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:50.211304 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:50.211753 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 02:17:50.212275 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":411,"threads_started":1}
I20260928 02:17:50.216193 8286 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8acd1af364f4ccab77157472aef21e0" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 35867 } }
I20260928 02:17:50.216766 8286 leader_election.cc:302] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8acd1af364f4ccab77157472aef21e0; no voters:
I20260928 02:17:50.217415 8286 leader_election.cc:288] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:17:50.217556 8291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:50.218264 8291 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0 [term 1 LEADER]: Becoming Leader. State: Replica: e8acd1af364f4ccab77157472aef21e0, State: Running, Role: LEADER
I20260928 02:17:50.218782 8291 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8acd1af364f4ccab77157472aef21e0" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 35867 } }
I20260928 02:17:50.219460 8286 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:17:50.221915 8293 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8acd1af364f4ccab77157472aef21e0. Latest consensus state: current_term: 1 leader_uuid: "e8acd1af364f4ccab77157472aef21e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8acd1af364f4ccab77157472aef21e0" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 35867 } } }
I20260928 02:17:50.222595 8293 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:50.222097 8292 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8acd1af364f4ccab77157472aef21e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8acd1af364f4ccab77157472aef21e0" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 35867 } } }
I20260928 02:17:50.222712 8292 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:50.223443 8300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:17:50.231184 8300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:17:50.240129 8300 catalog_manager.cc:1347] Generated new cluster ID: 9c499f51857c44f0a93205c1d51f9649
I20260928 02:17:50.240254 8300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:17:50.278440 8300 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:17:50.280830 8300 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:17:50.289391 8300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8acd1af364f4ccab77157472aef21e0: Generated new TSK 0
I20260928 02:17:50.290484 8300 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
I20260928 02:17:50.352219 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:35867
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:50.682669 8312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:50.682953 8312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:50.683107 8312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:50.690524 8312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:50.690646 8312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:50.690836 8312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:17:50.700963 8312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:35867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:50.702409 8312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:50.703682 8312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:50.717610 8317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:50.717546 8318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:50.718036 8320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:50.718369 8312 server_base.cc:1034] running on GCE node
I20260928 02:17:50.718963 8312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:50.720135 8312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:50.721343 8312 hybrid_clock.cc:648] HybridClock initialized: now 1790561870721312 us; error 27 us; skew 500 ppm
I20260928 02:17:50.721699 8312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:50 dist-test-slave-h37r krb5kdc[8206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561870, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:50.731309 8312 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:17:50.732990 8312 webserver.cc:460] Webserver started at http://127.4.140.193:41559/ using document root <none> and password file <none>
I20260928 02:17:50.733667 8312 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:50.733764 8312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:50.734018 8312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:50.736989 8312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "3896b270974f4105ad955afdd2e0d5ca"
format_stamp: "Formatted at 2026-09-28 02:17:50 on dist-test-slave-h37r"
server_key: "8c7cf8a651437fffdc2abcc84251fe47"
server_key_iv: "f86502eaa7183fd0a7a7f8e6bb820d2d"
server_key_version: "encryptionkey@0"
I20260928 02:17:50.737749 8312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "3896b270974f4105ad955afdd2e0d5ca"
format_stamp: "Formatted at 2026-09-28 02:17:50 on dist-test-slave-h37r"
server_key: "8c7cf8a651437fffdc2abcc84251fe47"
server_key_iv: "f86502eaa7183fd0a7a7f8e6bb820d2d"
server_key_version: "encryptionkey@0"
I20260928 02:17:50.743266 8312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:17:50.746990 8327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:50.748558 8312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:17:50.748740 8312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "3896b270974f4105ad955afdd2e0d5ca"
format_stamp: "Formatted at 2026-09-28 02:17:50 on dist-test-slave-h37r"
server_key: "8c7cf8a651437fffdc2abcc84251fe47"
server_key_iv: "f86502eaa7183fd0a7a7f8e6bb820d2d"
server_key_version: "encryptionkey@0"
I20260928 02:17:50.748960 8312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:50.763394 8312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:50.768404 8312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:50.768651 8312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:50.769510 8312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:50.770748 8312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:50.770816 8312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:50.770906 8312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:50.770952 8312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:50.804548 8312 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:32979
I20260928 02:17:50.804636 8439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:32979 every 8 connection(s)
I20260928 02:17:50.805824 8312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:17:50.813301 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 8312
I20260928 02:17:50.813517 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:17:50.814148 4659 external_mini_cluster.cc:1428] Setting key a656d28c7b6955d5f60096e2687bd46d
Sep 28 02:17:50 dist-test-slave-h37r krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561870, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:50.828372 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:50.808706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:58013 (local address 127.4.140.254:35867)
0928 02:17:50.809229 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:50.809234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:50.810251 (+ 1017us) server_negotiation.cc:407] Connection header received
0928 02:17:50.812105 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:50.812111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:50.812210 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:50.812327 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:50.813947 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:50.814774 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:50.816971 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:50.817363 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:50.817529 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:50.822093 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:50.822099 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:50.822103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:50.822136 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:50.824943 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:50.825719 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:50.825725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:50.825729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:50.825861 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:50.826442 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:50.826446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:50.826449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:50.826720 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:50.826891 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:50.827655 (+ 764us) server_negotiation.cc:299] Negotiation successful
0928 02:17:50.827922 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":229,"threads_started":1}
I20260928 02:17:50.829383 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:50.809380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:35867 (local address 127.4.140.193:58013)
0928 02:17:50.810061 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:50.810107 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:50.811793 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:50.812557 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:50.812565 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:50.813107 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:50.813778 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:50.813797 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:50.814967 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:50.814973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:50.816789 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:50.816813 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:50.817654 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:50.817662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:50.817997 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:50.818780 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:50.818814 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:50.821828 (+ 3014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:50.825092 (+ 3264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:50.825103 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:50.825122 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:50.825581 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:50.826009 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:50.826015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:50.826020 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:50.826262 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:50.826865 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:50.826875 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:50.827247 (+ 372us) client_negotiation.cc:769] Sending connection context
0928 02:17:50.827538 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 02:17:50.828135 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":261,"threads_started":1}
I20260928 02:17:50.831163 8440 heartbeater.cc:347] Connected to a master server at 127.4.140.254:35867
I20260928 02:17:50.831514 8440 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:50.832377 8440 heartbeater.cc:511] Master 127.4.140.254:35867 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.140.194@KRBTEST.COM; defaulting to no policy
I20260928 02:17:50.834707 8251 ts_manager.cc:194] Registered new tserver with Master: 3896b270974f4105ad955afdd2e0d5ca (127.4.140.193:32979)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.194@KRBTEST.COM" created.
I20260928 02:17:50.836822 8251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:58013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
I20260928 02:17:50.860307 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:35867
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:51.200254 8446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:51.200547 8446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:51.200642 8446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:51.207979 8446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:51.208093 8446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:51.208206 8446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:17:51.218205 8446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:35867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:51.219472 8446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:51.221098 8446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:51.234205 8451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:51.234246 8454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:51.234180 8452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:51.235019 8446 server_base.cc:1034] running on GCE node
I20260928 02:17:51.235677 8446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:51.236670 8446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:51.237902 8446 hybrid_clock.cc:648] HybridClock initialized: now 1790561871237861 us; error 54 us; skew 500 ppm
I20260928 02:17:51.238262 8446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:51 dist-test-slave-h37r krb5kdc[8206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561871, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:51.241513 8446 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:17:51.243085 8446 webserver.cc:460] Webserver started at http://127.4.140.194:40937/ using document root <none> and password file <none>
I20260928 02:17:51.243769 8446 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:51.243961 8446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:51.244287 8446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:51.247074 8446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "ba5d6cc7c1dc4fa7984e712c44ac7e87"
format_stamp: "Formatted at 2026-09-28 02:17:51 on dist-test-slave-h37r"
server_key: "af6a5f73c1119c2f2816bbb789398e4f"
server_key_iv: "889584e506dd694e57296e624bc84d5a"
server_key_version: "encryptionkey@0"
I20260928 02:17:51.247834 8446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "ba5d6cc7c1dc4fa7984e712c44ac7e87"
format_stamp: "Formatted at 2026-09-28 02:17:51 on dist-test-slave-h37r"
server_key: "af6a5f73c1119c2f2816bbb789398e4f"
server_key_iv: "889584e506dd694e57296e624bc84d5a"
server_key_version: "encryptionkey@0"
I20260928 02:17:51.252812 8446 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 02:17:51.256454 8461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:51.257787 8446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 02:17:51.257966 8446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "ba5d6cc7c1dc4fa7984e712c44ac7e87"
format_stamp: "Formatted at 2026-09-28 02:17:51 on dist-test-slave-h37r"
server_key: "af6a5f73c1119c2f2816bbb789398e4f"
server_key_iv: "889584e506dd694e57296e624bc84d5a"
server_key_version: "encryptionkey@0"
I20260928 02:17:51.258164 8446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:51.272743 8446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:51.277200 8446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:51.277496 8446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:51.278417 8446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:51.279743 8446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:51.279846 8446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:51.280017 8446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:51.280097 8446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:51.314249 8446 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:45099
I20260928 02:17:51.314284 8573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:45099 every 8 connection(s)
I20260928 02:17:51.315735 8446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:17:51.319998 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 8446
I20260928 02:17:51.320171 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:17:51.320686 4659 external_mini_cluster.cc:1428] Setting key 85407559eb3bb605023c919da313a465
Sep 28 02:17:51 dist-test-slave-h37r krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561871, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:51.337375 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:51.318489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:53917 (local address 127.4.140.254:35867)
0928 02:17:51.318670 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:51.318674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:51.320229 (+ 1555us) server_negotiation.cc:407] Connection header received
0928 02:17:51.322576 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:51.322581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:51.322659 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:51.322778 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:51.324479 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:51.325265 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:51.326823 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:51.327241 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:51.327378 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:51.331022 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:51.331028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:51.331031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:51.331062 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:51.333961 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:51.334792 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:51.334805 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:51.334808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:51.334945 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:51.335473 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:51.335479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:51.335482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:51.335783 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:51.336001 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:51.336724 (+ 723us) server_negotiation.cc:299] Negotiation successful
0928 02:17:51.336965 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.4.140.195@KRBTEST.COM; defaulting to no policy
I20260928 02:17:51.338524 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:51.318962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:35867 (local address 127.4.140.194:53917)
0928 02:17:51.320021 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:51.320092 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:51.322263 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:51.323035 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:51.323043 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:51.323557 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:51.324308 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:51.324335 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:51.325429 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:51.325435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:51.326675 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:51.326686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:51.327475 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:51.327482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:51.327721 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:51.328516 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:51.328538 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:51.330791 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:51.334117 (+ 3326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:51.334128 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:51.334147 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:51.334651 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:51.335124 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:51.335130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:51.335134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:51.335343 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:51.335950 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:51.335959 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:51.336337 (+ 378us) client_negotiation.cc:769] Sending connection context
0928 02:17:51.336621 (+ 284us) client_negotiation.cc:240] Negotiation successful
0928 02:17:51.337218 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":516,"threads_started":1}
I20260928 02:17:51.340111 8574 heartbeater.cc:347] Connected to a master server at 127.4.140.254:35867
I20260928 02:17:51.340521 8574 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.195@KRBTEST.COM" created.
I20260928 02:17:51.341596 8574 heartbeater.cc:511] Master 127.4.140.254:35867 requested a full tablet report, sending...
I20260928 02:17:51.343310 8251 ts_manager.cc:194] Registered new tserver with Master: ba5d6cc7c1dc4fa7984e712c44ac7e87 (127.4.140.194:45099)
I20260928 02:17:51.344230 8251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:53917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
I20260928 02:17:51.368214 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:35867
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:51.686743 8579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:51.687059 8579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:51.687228 8579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:51.694404 8579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:51.694550 8579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:51.694726 8579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:17:51.704387 8579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:35867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:51.705654 8579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:51.706924 8579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:51.719949 8584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:51.719950 8587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:51.719988 8585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:51.720610 8579 server_base.cc:1034] running on GCE node
I20260928 02:17:51.721161 8579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:51.721983 8579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:51.723172 8579 hybrid_clock.cc:648] HybridClock initialized: now 1790561871723133 us; error 31 us; skew 500 ppm
I20260928 02:17:51.723472 8579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:51 dist-test-slave-h37r krb5kdc[8206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561871, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:51.726621 8579 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:17:51.728109 8579 webserver.cc:460] Webserver started at http://127.4.140.195:44199/ using document root <none> and password file <none>
I20260928 02:17:51.728760 8579 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:51.728893 8579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:51.729192 8579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:51.732005 8579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "7f51faa1b3f34e1a8984f3f772c7818f"
format_stamp: "Formatted at 2026-09-28 02:17:51 on dist-test-slave-h37r"
server_key: "7a0f56e770e36dca5520ed4d685ae36f"
server_key_iv: "6a892ea53e0ea29e6642dd3877fc7dd2"
server_key_version: "encryptionkey@0"
I20260928 02:17:51.732769 8579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "7f51faa1b3f34e1a8984f3f772c7818f"
format_stamp: "Formatted at 2026-09-28 02:17:51 on dist-test-slave-h37r"
server_key: "7a0f56e770e36dca5520ed4d685ae36f"
server_key_iv: "6a892ea53e0ea29e6642dd3877fc7dd2"
server_key_version: "encryptionkey@0"
I20260928 02:17:51.738237 8579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:17:51.741575 8594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:51.742784 8579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:17:51.742959 8579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "7f51faa1b3f34e1a8984f3f772c7818f"
format_stamp: "Formatted at 2026-09-28 02:17:51 on dist-test-slave-h37r"
server_key: "7a0f56e770e36dca5520ed4d685ae36f"
server_key_iv: "6a892ea53e0ea29e6642dd3877fc7dd2"
server_key_version: "encryptionkey@0"
I20260928 02:17:51.743152 8579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:51.761369 8579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:51.766036 8579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:51.766314 8579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:51.767261 8579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:51.768632 8579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:51.768733 8579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:51.768886 8579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:51.768965 8579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:51.802520 8579 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:40157
I20260928 02:17:51.802572 8706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:40157 every 8 connection(s)
I20260928 02:17:51.803840 8579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:17:51.807063 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 8579
I20260928 02:17:51.807236 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:17:51.807739 4659 external_mini_cluster.cc:1428] Setting key 50257ccd5ac947e07f0ac7674270c945
Sep 28 02:17:51 dist-test-slave-h37r krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561871, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:51.824952 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:51.806623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:42143 (local address 127.4.140.254:35867)
0928 02:17:51.806852 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:51.806860 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:51.808679 (+ 1819us) server_negotiation.cc:407] Connection header received
0928 02:17:51.810928 (+ 2249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:51.810932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:51.810999 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:51.811086 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:51.812640 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:51.813392 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:51.814859 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:51.815268 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:51.815369 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:51.818764 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:51.818769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:51.818772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:51.818802 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:51.821574 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:51.822344 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:51.822352 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:51.822356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:51.822537 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:51.822998 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:51.823003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:51.823007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:51.823345 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:51.823510 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:51.824382 (+ 872us) server_negotiation.cc:299] Negotiation successful
0928 02:17:51.824589 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 02:17:51.825977 8709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:51.807317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:35867 (local address 127.4.140.195:42143)
0928 02:17:51.808228 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:51.808550 (+ 322us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:51.810617 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:51.811292 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:51.811300 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:51.811803 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:51.812487 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:51.812505 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:51.813528 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:51.813532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:51.814734 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:51.814746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:51.815465 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:51.815469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:51.815655 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:51.816379 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:51.816407 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:51.818582 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:51.821697 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:51.821708 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:51.821725 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:51.822234 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:51.822657 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:51.822660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:51.822662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:51.822898 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:51.823478 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:51.823487 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:51.823884 (+ 397us) client_negotiation.cc:769] Sending connection context
0928 02:17:51.824285 (+ 401us) client_negotiation.cc:240] Negotiation successful
0928 02:17:51.824831 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":345,"threads_started":1}
I20260928 02:17:51.827237 8707 heartbeater.cc:347] Connected to a master server at 127.4.140.254:35867
I20260928 02:17:51.827498 8707 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:51.828212 8707 heartbeater.cc:511] Master 127.4.140.254:35867 requested a full tablet report, sending...
I20260928 02:17:51.829515 8251 ts_manager.cc:194] Registered new tserver with Master: 7f51faa1b3f34e1a8984f3f772c7818f (127.4.140.195:40157)
I20260928 02:17:51.830309 8251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:42143
I20260928 02:17:51.835531 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:17:51.839866 8440 heartbeater.cc:503] Master 127.4.140.254:35867 was elected leader, sending a full tablet report...
Sep 28 02:17:51 dist-test-slave-h37r krb5kdc[8206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:17:51 dist-test-slave-h37r krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:17:51.885136 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:51.870115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49612 (local address 127.4.140.193:32979)
0928 02:17:51.870785 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:51.870793 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:51.870819 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:17:51.870879 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:51.870885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:51.871103 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:51.871233 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:51.872656 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:51.873704 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:51.874977 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:51.875564 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:51.875676 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:51.879678 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:51.879690 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:51.879716 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:51.879756 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:51.882489 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:51.883110 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:51.883116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:51.883118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:51.883237 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:51.883583 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:51.883587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:51.883590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:51.884057 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:51.884275 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:51.884540 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 02:17:51.884734 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":385,"threads_started":1}
Sep 28 02:17:51 dist-test-slave-h37r krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:51.902225 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:51.889263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33706 (local address 127.4.140.254:35867)
0928 02:17:51.889455 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:51.889461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:51.889619 (+ 158us) server_negotiation.cc:407] Connection header received
0928 02:17:51.889803 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:51.889813 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:51.889874 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:51.889975 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:51.891204 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:51.891975 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:51.893164 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:51.893578 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:51.893709 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:51.897538 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:51.897542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:51.897545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:51.897579 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:51.900133 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:51.900694 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:51.900698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:51.900700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:51.900814 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:51.901098 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:51.901101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:51.901103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:51.901396 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:51.901567 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:51.901826 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 02:17:51.901992 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 02:17:51.902802 8251 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33706
I20260928 02:17:51.903854 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 8312
I20260928 02:17:51.916765 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:51.904911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.4.140.193:32979)
0928 02:17:51.905129 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:51.905142 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:51.905197 (+ 55us) server_negotiation.cc:407] Connection header received
0928 02:17:51.905349 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:51.905352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:51.905414 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:51.905490 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:51.906652 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:51.907387 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:51.908847 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:51.909269 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:51.909364 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:51.911829 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:51.911833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:51.911836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:51.911865 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:51.914594 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:51.915181 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:51.915185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:51.915187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:51.915300 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:51.915590 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:51.915593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:51.915595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:51.915820 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:51.916078 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:51.916257 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 02:17:51.916406 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 02:17:51.917364 8435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:52.003582 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 8312
I20260928 02:17:52.020731 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 8446
Sep 28 02:17:52 dist-test-slave-h37r krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.194@KRBTEST.COM
I20260928 02:17:52.037150 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:52.021813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.4.140.194:45099)
0928 02:17:52.022405 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:52.022412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:52.022438 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:17:52.022506 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:52.022512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:52.022735 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:52.022868 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:52.024216 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:52.025355 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:52.026640 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:52.027188 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:52.027284 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:52.031485 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:52.031496 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:52.031511 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:52.031545 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:52.034324 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:52.035012 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:52.035019 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:52.035021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:52.035149 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:52.035530 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:52.035536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:52.035539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:52.035875 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:52.036247 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:52.036548 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 02:17:52.036799 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":294,"threads_started":1}
I20260928 02:17:52.038388 8569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:52.120985 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 8446
I20260928 02:17:52.137856 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 8579
Sep 28 02:17:52 dist-test-slave-h37r krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.195@KRBTEST.COM
I20260928 02:17:52.154464 8721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:52.138824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.4.140.195:40157)
0928 02:17:52.139439 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:52.139446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:52.139475 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:17:52.139535 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:52.139542 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:52.139749 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:52.139956 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:52.141147 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:52.142177 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:52.143532 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:52.144289 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:52.144401 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:52.148861 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:52.148872 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:52.148887 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:52.148923 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:52.151713 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:52.152464 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:52.152471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:52.152473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:52.152602 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:52.153040 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:52.153043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:52.153045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:52.153369 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:52.153598 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:52.153876 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 02:17:52.154076 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":322,"threads_started":1}
I20260928 02:17:52.155695 8702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:52.240233 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 8579
I20260928 02:17:52.255858 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 8219
I20260928 02:17:52.257076 8281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:52.361274 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 8219
2026-09-28T02:17:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3937 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:17:52 dist-test-slave-h37r krb5kdc[8726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:17:52 dist-test-slave-h37r krb5kdc[8726](info): set up 2 sockets
Sep 28 02:17:52 dist-test-slave-h37r krb5kdc[8726](info): commencing operation
krb5kdc: starting...
W20260928 02:17:53.452433 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:17:53 dist-test-slave-h37r krb5kdc[8726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:17:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:17:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:17:53.654973 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:43149
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:43149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:53.986817 8739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:53.987118 8739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:53.987282 8739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:53.994642 8739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:17:53.994760 8739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:53.994848 8739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:53.994953 8739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:17:53.995023 8739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:17:54.005330 8739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:43149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:43149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:54.006700 8739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:54.008059 8739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:54.021027 8747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:54.020955 8745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:54.020984 8744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:54.022648 8739 server_base.cc:1034] running on GCE node
I20260928 02:17:54.023525 8739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:54.025202 8739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:54.026408 8739 hybrid_clock.cc:648] HybridClock initialized: now 1790561874026366 us; error 34 us; skew 500 ppm
I20260928 02:17:54.026783 8739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:54 dist-test-slave-h37r krb5kdc[8726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561874, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:54.029908 8739 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:17:54.031143 8739 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:54.031235 8739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:54.031507 8739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:54.034535 8739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "6bb9d6bd76cc40819323259c7e942c68"
format_stamp: "Formatted at 2026-09-28 02:17:54 on dist-test-slave-h37r"
server_key: "5d0c03a2baf0f7931951d1fd2c5d7ec2"
server_key_iv: "17d3eb0eeb671af09fd9a68b10f5cc2e"
server_key_version: "encryptionkey@0"
I20260928 02:17:54.035282 8739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "6bb9d6bd76cc40819323259c7e942c68"
format_stamp: "Formatted at 2026-09-28 02:17:54 on dist-test-slave-h37r"
server_key: "5d0c03a2baf0f7931951d1fd2c5d7ec2"
server_key_iv: "17d3eb0eeb671af09fd9a68b10f5cc2e"
server_key_version: "encryptionkey@0"
I20260928 02:17:54.040503 8739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:17:54.044373 8753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:54.045805 8739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:17:54.046013 8739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "6bb9d6bd76cc40819323259c7e942c68"
format_stamp: "Formatted at 2026-09-28 02:17:54 on dist-test-slave-h37r"
server_key: "5d0c03a2baf0f7931951d1fd2c5d7ec2"
server_key_iv: "17d3eb0eeb671af09fd9a68b10f5cc2e"
server_key_version: "encryptionkey@0"
I20260928 02:17:54.046252 8739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:54.062528 8739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:54.066810 8739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:54.067103 8739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:54.087745 8739 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:43149
I20260928 02:17:54.087776 8804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:43149 every 8 connection(s)
I20260928 02:17:54.088825 8739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:17:54.093050 8805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:54.094012 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 8739
I20260928 02:17:54.094247 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:17:54.095188 4659 external_mini_cluster.cc:1428] Setting key 7726298890daddb9337bfbd7067754e8
I20260928 02:17:54.109261 8805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68: Bootstrap starting.
Sep 28 02:17:54 dist-test-slave-h37r krb5kdc[8726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:54.113337 8805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:54.114706 8805 log.cc:824] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:54.117971 8805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68: No bootstrap required, opened a new log
I20260928 02:17:54.119799 8808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:54.097454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52586 (local address 127.4.140.254:43149)
0928 02:17:54.098321 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:54.098333 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:54.098380 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:17:54.100128 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:54.100153 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:54.100600 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:54.100963 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:54.102962 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:54.104377 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:54.106419 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:54.107200 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:54.107344 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:54.112312 (+ 4968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:54.112321 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:54.112337 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:54.112378 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:54.115707 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:54.116466 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:54.116479 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:54.116485 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:54.116651 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:54.117038 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:54.117044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:54.117047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:54.117598 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:54.117911 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:54.118356 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 02:17:54.118796 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":414,"threads_started":1}
I20260928 02:17:54.127113 8805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb9d6bd76cc40819323259c7e942c68" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43149 } }
I20260928 02:17:54.127583 8805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:54.127698 8805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb9d6bd76cc40819323259c7e942c68, State: Initialized, Role: FOLLOWER
I20260928 02:17:54.128378 8805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb9d6bd76cc40819323259c7e942c68" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43149 } }
I20260928 02:17:54.128669 8805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:17:54.128846 8805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:17:54.129055 8805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:54.131656 8805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb9d6bd76cc40819323259c7e942c68" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43149 } }
I20260928 02:17:54.132238 8805 leader_election.cc:302] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bb9d6bd76cc40819323259c7e942c68; no voters:
I20260928 02:17:54.132908 8805 leader_election.cc:288] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:17:54.133062 8810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:54.134295 8810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68 [term 1 LEADER]: Becoming Leader. State: Replica: 6bb9d6bd76cc40819323259c7e942c68, State: Running, Role: LEADER
I20260928 02:17:54.134958 8810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb9d6bd76cc40819323259c7e942c68" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43149 } }
I20260928 02:17:54.135550 8805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:17:54.138504 8811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bb9d6bd76cc40819323259c7e942c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb9d6bd76cc40819323259c7e942c68" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43149 } } }
I20260928 02:17:54.138566 8812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bb9d6bd76cc40819323259c7e942c68. Latest consensus state: current_term: 1 leader_uuid: "6bb9d6bd76cc40819323259c7e942c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb9d6bd76cc40819323259c7e942c68" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43149 } } }
I20260928 02:17:54.140072 8812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68 [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:54.144025 8811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68 [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:54.149183 8819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:17:54.154558 8819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:17:54.163856 8819 catalog_manager.cc:1347] Generated new cluster ID: 9dee7ecb1b6047c99aeadd238e3784de
I20260928 02:17:54.164019 8819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:17:54.194703 8819 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:17:54.196710 8819 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:17:54.211541 8819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68: Generated new TSK 0
I20260928 02:17:54.212512 8819 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
I20260928 02:17:54.264043 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43149
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:54.596114 8831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:54.596426 8831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:54.596521 8831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:54.604041 8831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:54.604161 8831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:54.604276 8831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:17:54.614185 8831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:54.615679 8831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:54.617077 8831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:54.629935 8837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:54.630038 8839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:54.630013 8836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:54.630960 8831 server_base.cc:1034] running on GCE node
I20260928 02:17:54.631541 8831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:54.632580 8831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:54.633829 8831 hybrid_clock.cc:648] HybridClock initialized: now 1790561874633791 us; error 58 us; skew 500 ppm
I20260928 02:17:54.634186 8831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:54 dist-test-slave-h37r krb5kdc[8726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561874, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:54.637557 8831 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:17:54.638769 8831 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:54.638927 8831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:54.639249 8831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:54.642241 8831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "006c8083331e428c88ec278c62a12c44"
format_stamp: "Formatted at 2026-09-28 02:17:54 on dist-test-slave-h37r"
server_key: "079528a40c4546738ae7938c18a08d6a"
server_key_iv: "23d758e6d77cc3468c2e677707678877"
server_key_version: "encryptionkey@0"
I20260928 02:17:54.643085 8831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "006c8083331e428c88ec278c62a12c44"
format_stamp: "Formatted at 2026-09-28 02:17:54 on dist-test-slave-h37r"
server_key: "079528a40c4546738ae7938c18a08d6a"
server_key_iv: "23d758e6d77cc3468c2e677707678877"
server_key_version: "encryptionkey@0"
I20260928 02:17:54.648581 8831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 02:17:54.652675 8845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:54.654091 8831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:17:54.654281 8831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "006c8083331e428c88ec278c62a12c44"
format_stamp: "Formatted at 2026-09-28 02:17:54 on dist-test-slave-h37r"
server_key: "079528a40c4546738ae7938c18a08d6a"
server_key_iv: "23d758e6d77cc3468c2e677707678877"
server_key_version: "encryptionkey@0"
I20260928 02:17:54.654527 8831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:54.695393 8831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:54.699936 8831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:54.700232 8831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:54.701216 8831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:54.702589 8831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:54.702704 8831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:54.702836 8831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:54.702929 8831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:54.736315 8831 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:46147
I20260928 02:17:54.736362 8957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:46147 every 8 connection(s)
I20260928 02:17:54.737201 8831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:17:54.744329 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 8831
I20260928 02:17:54.744534 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:17:54.745347 4659 external_mini_cluster.cc:1428] Setting key 2dbf028e266f6c59a0cdb9a6328aa740
Sep 28 02:17:54 dist-test-slave-h37r krb5kdc[8726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561874, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:54.761477 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:54.739729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:35065 (local address 127.4.140.254:43149)
0928 02:17:54.740199 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:54.740203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:54.741264 (+ 1061us) server_negotiation.cc:407] Connection header received
0928 02:17:54.743093 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:54.743098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:54.743202 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:54.743322 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:54.745483 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:54.746405 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:54.748485 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:54.748945 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:54.749043 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:54.753432 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:54.753437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:54.753441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:54.753473 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:54.756728 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:54.757837 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:54.757843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:54.757846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:54.758003 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:54.758568 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:54.758573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:54.758575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:54.758999 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:54.759182 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:54.760731 (+ 1549us) server_negotiation.cc:299] Negotiation successful
0928 02:17:54.760996 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":256,"threads_started":1}
I20260928 02:17:54.762199 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:54.740350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43149 (local address 127.4.140.193:35065)
0928 02:17:54.741097 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:54.741135 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:54.742812 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:54.743571 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:54.743579 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:54.744376 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:54.745238 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:54.745262 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:54.746597 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:54.746601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:54.748291 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:54.748309 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:54.749190 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:54.749199 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:54.749544 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:54.750355 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:54.750390 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:54.753173 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:54.756920 (+ 3747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:54.756929 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:54.756948 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:54.757704 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:54.758128 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:54.758133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:54.758137 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:54.758426 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:54.759161 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:54.759169 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:54.759697 (+ 528us) client_negotiation.cc:769] Sending connection context
0928 02:17:54.760076 (+ 379us) client_negotiation.cc:240] Negotiation successful
0928 02:17:54.760704 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":287,"threads_started":1}
I20260928 02:17:54.763924 8958 heartbeater.cc:347] Connected to a master server at 127.4.140.254:43149
I20260928 02:17:54.764259 8958 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:54.764930 8958 heartbeater.cc:511] Master 127.4.140.254:43149 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.140.194@KRBTEST.COM; defaulting to no policy
I20260928 02:17:54.767133 8770 ts_manager.cc:194] Registered new tserver with Master: 006c8083331e428c88ec278c62a12c44 (127.4.140.193:46147)
I20260928 02:17:54.768802 8770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:35065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
I20260928 02:17:54.796581 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43149
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:55.122970 8964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:55.123258 8964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:55.123431 8964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:55.130960 8964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:55.131074 8964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:55.131264 8964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:17:55.141260 8964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:55.142648 8964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:55.144081 8964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:55.157672 8970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:55.157768 8969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:55.158150 8972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:55.158897 8964 server_base.cc:1034] running on GCE node
I20260928 02:17:55.159512 8964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:55.160583 8964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:55.161787 8964 hybrid_clock.cc:648] HybridClock initialized: now 1790561875161724 us; error 50 us; skew 500 ppm
I20260928 02:17:55.162122 8964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:55 dist-test-slave-h37r krb5kdc[8726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561875, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:55.170894 8964 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:17:55.172281 8964 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:55.172415 8964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:55.172780 8964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:55.176044 8964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "d2efe484a0434b7298bc751b0ae6239f"
format_stamp: "Formatted at 2026-09-28 02:17:55 on dist-test-slave-h37r"
server_key: "8a5554d79055277d1d7e6ac8022f0fef"
server_key_iv: "0ea42d9576235a6dabb4aa10b81a94b7"
server_key_version: "encryptionkey@0"
I20260928 02:17:55.176939 8964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "d2efe484a0434b7298bc751b0ae6239f"
format_stamp: "Formatted at 2026-09-28 02:17:55 on dist-test-slave-h37r"
server_key: "8a5554d79055277d1d7e6ac8022f0fef"
server_key_iv: "0ea42d9576235a6dabb4aa10b81a94b7"
server_key_version: "encryptionkey@0"
I20260928 02:17:55.182554 8964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 02:17:55.186193 8978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:55.187512 8964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:17:55.187702 8964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "d2efe484a0434b7298bc751b0ae6239f"
format_stamp: "Formatted at 2026-09-28 02:17:55 on dist-test-slave-h37r"
server_key: "8a5554d79055277d1d7e6ac8022f0fef"
server_key_iv: "0ea42d9576235a6dabb4aa10b81a94b7"
server_key_version: "encryptionkey@0"
I20260928 02:17:55.187978 8964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:55.208448 8964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:55.212766 8964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:55.213053 8964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:55.214039 8964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:55.215405 8964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:55.215524 8964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:55.215663 8964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:55.215750 8964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:55.249647 8964 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:43519
I20260928 02:17:55.249701 9090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:43519 every 8 connection(s)
I20260928 02:17:55.250523 8964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:17:55.256906 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 8964
I20260928 02:17:55.257078 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:17:55.257623 4659 external_mini_cluster.cc:1428] Setting key a07f7efdba7f0d57375440e2280525c5
Sep 28 02:17:55 dist-test-slave-h37r krb5kdc[8726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561875, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:55.271106 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:55.253059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:59581 (local address 127.4.140.254:43149)
0928 02:17:55.253260 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:55.253266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:55.254587 (+ 1321us) server_negotiation.cc:407] Connection header received
0928 02:17:55.256410 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:55.256414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:55.256486 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:55.256573 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:55.258603 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.259522 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.261110 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.261548 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.261684 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:55.265209 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:55.265214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:55.265217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:55.265247 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:55.267825 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:55.268767 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:55.268771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:55.268773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:55.268900 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:55.269378 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:55.269381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:55.269383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:55.269626 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:55.269729 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:55.270690 (+ 961us) server_negotiation.cc:299] Negotiation successful
0928 02:17:55.270861 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:17:55.272639 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:55.253643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43149 (local address 127.4.140.194:59581)
0928 02:17:55.254419 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:55.254462 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:55.256140 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:55.256843 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:55.256856 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:55.257678 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:55.258433 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:55.258453 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.259661 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.259667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:55.260947 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:55.260968 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.261771 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.261778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:55.262004 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:55.262699 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:55.262720 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:55.264986 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:55.267991 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:55.268003 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:55.268024 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:55.268635 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:55.269015 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:55.269023 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:55.269029 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:55.269265 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:55.269758 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:55.269769 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:55.270211 (+ 442us) client_negotiation.cc:769] Sending connection context
0928 02:17:55.270528 (+ 317us) client_negotiation.cc:240] Negotiation successful
0928 02:17:55.271110 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":315,"threads_started":1}
I20260928 02:17:55.274231 9091 heartbeater.cc:347] Connected to a master server at 127.4.140.254:43149
I20260928 02:17:55.274588 9091 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:55.275235 9091 heartbeater.cc:511] Master 127.4.140.254:43149 requested a full tablet report, sending...
I20260928 02:17:55.276752 8770 ts_manager.cc:194] Registered new tserver with Master: d2efe484a0434b7298bc751b0ae6239f (127.4.140.194:43519)
WARNING: no policy specified for kudu/127.4.140.195@KRBTEST.COM; defaulting to no policy
I20260928 02:17:55.277626 8770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:59581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
I20260928 02:17:55.303807 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43149
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:17:55.622498 9096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:55.622828 9096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:55.623006 9096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:55.630385 9096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:55.630556 9096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:55.630736 9096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:17:55.640672 9096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:55.642035 9096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:55.643358 9096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:55.656651 9104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:55.656690 9101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:55.656651 9102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:55.657964 9096 server_base.cc:1034] running on GCE node
I20260928 02:17:55.658646 9096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:55.659714 9096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:55.660952 9096 hybrid_clock.cc:648] HybridClock initialized: now 1790561875660898 us; error 48 us; skew 500 ppm
I20260928 02:17:55.661279 9096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:17:55 dist-test-slave-h37r krb5kdc[8726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561875, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:17:55.664486 9096 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:17:55.665761 9096 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:55.665851 9096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:55.666097 9096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:55.669039 9096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "7a937508c9864d22b0a2b43949df43df"
format_stamp: "Formatted at 2026-09-28 02:17:55 on dist-test-slave-h37r"
server_key: "f406dec627d24667efe618c23307923b"
server_key_iv: "ac6d6dd29af015d152cc2856d980c20c"
server_key_version: "encryptionkey@0"
I20260928 02:17:55.669788 9096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "7a937508c9864d22b0a2b43949df43df"
format_stamp: "Formatted at 2026-09-28 02:17:55 on dist-test-slave-h37r"
server_key: "f406dec627d24667efe618c23307923b"
server_key_iv: "ac6d6dd29af015d152cc2856d980c20c"
server_key_version: "encryptionkey@0"
I20260928 02:17:55.675143 9096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:17:55.678798 9110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:55.680406 9096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:17:55.680612 9096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "7a937508c9864d22b0a2b43949df43df"
format_stamp: "Formatted at 2026-09-28 02:17:55 on dist-test-slave-h37r"
server_key: "f406dec627d24667efe618c23307923b"
server_key_iv: "ac6d6dd29af015d152cc2856d980c20c"
server_key_version: "encryptionkey@0"
I20260928 02:17:55.680850 9096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:55.704936 9096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:55.709515 9096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:55.709802 9096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:55.710799 9096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:55.712174 9096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:55.712285 9096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:55.712421 9096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:55.712494 9096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:55.745720 9096 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:44153
I20260928 02:17:55.745813 9222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:44153 every 8 connection(s)
I20260928 02:17:55.746682 9096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:17:55.752637 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 9096
I20260928 02:17:55.752914 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:17:55.753561 4659 external_mini_cluster.cc:1428] Setting key de2cf4ec0df86c4dc5cc32e8192db811
Sep 28 02:17:55 dist-test-slave-h37r krb5kdc[8726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561875, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:17:55.766489 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:55.749151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:51987 (local address 127.4.140.254:43149)
0928 02:17:55.749371 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:55.749378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:55.750690 (+ 1312us) server_negotiation.cc:407] Connection header received
0928 02:17:55.752917 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:55.752937 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:55.753136 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:55.753227 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:55.755016 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.755777 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.757331 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.757754 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.757900 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:55.761218 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:55.761223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:55.761226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:55.761257 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:55.763598 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:55.764516 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:55.764520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:55.764523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:55.764643 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:55.765062 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:55.765065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:55.765067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:55.765313 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:55.765419 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:55.766083 (+ 664us) server_negotiation.cc:299] Negotiation successful
0928 02:17:55.766238 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:17:55.767774 9225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:55.749752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43149 (local address 127.4.140.195:51987)
0928 02:17:55.750532 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:55.750571 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:55.752495 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:55.753493 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:55.753504 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:55.754232 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:55.754796 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:55.754847 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.755949 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.755954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:55.757189 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:55.757200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.757992 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.758001 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:55.758225 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:55.758890 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:17:55.758912 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:55.761030 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:55.763747 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:55.763755 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:55.763770 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:55.764345 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:55.764764 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:17:55.764768 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:17:55.764771 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:17:55.764946 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:17:55.765428 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:55.765434 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:55.765756 (+ 322us) client_negotiation.cc:769] Sending connection context
0928 02:17:55.766000 (+ 244us) client_negotiation.cc:240] Negotiation successful
0928 02:17:55.766396 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":303,"threads_started":1}
I20260928 02:17:55.769291 9223 heartbeater.cc:347] Connected to a master server at 127.4.140.254:43149
I20260928 02:17:55.769588 9223 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:55.770227 9223 heartbeater.cc:511] Master 127.4.140.254:43149 requested a full tablet report, sending...
I20260928 02:17:55.771641 8769 ts_manager.cc:194] Registered new tserver with Master: 7a937508c9864d22b0a2b43949df43df (127.4.140.195:44153)
I20260928 02:17:55.772109 8958 heartbeater.cc:503] Master 127.4.140.254:43149 was elected leader, sending a full tablet report...
I20260928 02:17:55.772810 8769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:51987
I20260928 02:17:55.781665 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:17:55.798198 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:55.786300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.4.140.254:43149)
0928 02:17:55.786568 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:55.786574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:55.786831 (+ 257us) server_negotiation.cc:407] Connection header received
0928 02:17:55.786965 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:55.786969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:55.787031 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:55.787109 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:55.788562 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.789294 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.790541 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.790994 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.791099 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:55.793419 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:55.793424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:55.793427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:55.793464 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:55.795837 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:55.796531 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:55.796534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:55.796537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:55.796657 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:55.796951 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:55.796954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:55.796956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:55.797252 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:55.797423 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:55.797692 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 02:17:55.797844 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:17:55.802243 8770 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:17:55.808987 8770 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:17:55.846627 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:55.821603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43290 (local address 127.4.140.195:44153)
0928 02:17:55.838302 (+ 16699us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:55.838312 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:55.838344 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:17:55.838421 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:55.838427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:55.838594 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:55.838954 (+ 360us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:55.839642 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.841379 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.844089 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.845437 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.845560 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:55.845796 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:55.846099 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 02:17:55.846274 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16488,"thread_start_us":16319,"threads_started":1}
I20260928 02:17:55.846755 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:55.821464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:44153 (local address 127.0.0.1:43290)
0928 02:17:55.822386 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:55.822434 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:55.822651 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:55.838803 (+ 16152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:55.838808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:55.838857 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:55.839412 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:55.839426 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.841514 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.841521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:55.843939 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:55.843951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.845621 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.845628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:55.845829 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:55.845884 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 02:17:55.846043 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 02:17:55.846258 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":34,"spinlock_wait_cycles":9600,"thread_start_us":406,"threads_started":1}
I20260928 02:17:55.846882 9239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:55.822691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:46147 (local address 127.0.0.1:37912)
0928 02:17:55.837884 (+ 15193us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:55.837904 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:55.838019 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:55.839049 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:55.839053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:55.839073 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:55.839582 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:55.839594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.841565 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.841569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:55.843679 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:55.843693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.845943 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.845947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:55.846174 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:55.846208 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:17:55.846293 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 02:17:55.846427 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15065,"mutex_wait_us":772,"thread_start_us":14361,"threads_started":1}
I20260928 02:17:55.847513 9240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:55.823464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46402 (local address 127.4.140.194:43519)
0928 02:17:55.836908 (+ 13444us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:55.836922 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:55.837075 (+ 153us) server_negotiation.cc:407] Connection header received
0928 02:17:55.838307 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:55.838316 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:55.838546 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:55.838788 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:55.840147 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.841104 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.844662 (+ 3558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.845653 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.845784 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:55.845964 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:55.847108 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0928 02:17:55.847288 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13179,"thread_start_us":12193,"threads_started":1}
I20260928 02:17:55.847576 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:55.823090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37912 (local address 127.4.140.193:46147)
0928 02:17:55.837007 (+ 13917us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:55.837019 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:55.838139 (+ 1120us) server_negotiation.cc:407] Connection header received
0928 02:17:55.838327 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:55.838338 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:55.838588 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:55.838787 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:55.839752 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.841400 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.843835 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.845410 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.845556 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:55.845798 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:55.847074 (+ 1276us) server_negotiation.cc:299] Negotiation successful
0928 02:17:55.847249 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13233,"thread_start_us":13034,"threads_started":1}
I20260928 02:17:55.846976 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:55.822690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:43519 (local address 127.0.0.1:46402)
0928 02:17:55.836879 (+ 14189us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:55.836906 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:55.837305 (+ 399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:55.838793 (+ 1488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:55.838800 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:55.838844 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:55.839993 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:55.840002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.841245 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.841251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:55.844526 (+ 3275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:55.844537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.846106 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.846111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:55.846290 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:55.846317 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 02:17:55.846397 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:17:55.846532 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14005,"mutex_wait_us":250,"spinlock_wait_cycles":1920,"thread_start_us":13869,"threads_started":1}
I20260928 02:17:55.851241 9026 tablet_service.cc:1467] Processing CreateTablet for tablet d86b06fe19fb45099139c740946ddd75 (DEFAULT_TABLE table=test-table [id=19e35ef318a44f349ed67efbfbf416ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:55.851260 9158 tablet_service.cc:1467] Processing CreateTablet for tablet d86b06fe19fb45099139c740946ddd75 (DEFAULT_TABLE table=test-table [id=19e35ef318a44f349ed67efbfbf416ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:55.851260 8893 tablet_service.cc:1467] Processing CreateTablet for tablet d86b06fe19fb45099139c740946ddd75 (DEFAULT_TABLE table=test-table [id=19e35ef318a44f349ed67efbfbf416ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:55.852680 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d86b06fe19fb45099139c740946ddd75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:55.852679 8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d86b06fe19fb45099139c740946ddd75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:55.852860 9026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d86b06fe19fb45099139c740946ddd75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:55.864212 9241 tablet_bootstrap.cc:492] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f: Bootstrap starting.
I20260928 02:17:55.867063 9242 tablet_bootstrap.cc:492] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44: Bootstrap starting.
I20260928 02:17:55.867808 9241 tablet_bootstrap.cc:654] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:55.867868 9243 tablet_bootstrap.cc:492] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df: Bootstrap starting.
I20260928 02:17:55.869259 9241 log.cc:824] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:55.871157 9242 tablet_bootstrap.cc:654] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:55.871287 9243 tablet_bootstrap.cc:654] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:55.872423 9242 log.cc:824] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:55.872498 9243 log.cc:824] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:55.873538 9241 tablet_bootstrap.cc:492] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f: No bootstrap required, opened a new log
I20260928 02:17:55.873842 9241 ts_tablet_manager.cc:1397] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 02:17:55.875293 9243 tablet_bootstrap.cc:492] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df: No bootstrap required, opened a new log
I20260928 02:17:55.875567 9243 ts_tablet_manager.cc:1397] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 02:17:55.875614 9242 tablet_bootstrap.cc:492] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44: No bootstrap required, opened a new log
I20260928 02:17:55.875862 9242 ts_tablet_manager.cc:1397] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 02:17:55.882551 9243 raft_consensus.cc:348] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a937508c9864d22b0a2b43949df43df" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44153 } } peers { permanent_uuid: "d2efe484a0434b7298bc751b0ae6239f" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43519 } } peers { permanent_uuid: "006c8083331e428c88ec278c62a12c44" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 46147 } }
I20260928 02:17:55.883107 9243 raft_consensus.cc:374] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:55.883239 9243 raft_consensus.cc:729] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a937508c9864d22b0a2b43949df43df, State: Initialized, Role: FOLLOWER
I20260928 02:17:55.883978 9243 consensus_queue.cc:260] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a937508c9864d22b0a2b43949df43df" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44153 } } peers { permanent_uuid: "d2efe484a0434b7298bc751b0ae6239f" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43519 } } peers { permanent_uuid: "006c8083331e428c88ec278c62a12c44" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 46147 } }
I20260928 02:17:55.884755 9241 raft_consensus.cc:348] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a937508c9864d22b0a2b43949df43df" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44153 } } peers { permanent_uuid: "d2efe484a0434b7298bc751b0ae6239f" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43519 } } peers { permanent_uuid: "006c8083331e428c88ec278c62a12c44" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 46147 } }
I20260928 02:17:55.885245 9241 raft_consensus.cc:374] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:55.885380 9241 raft_consensus.cc:729] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2efe484a0434b7298bc751b0ae6239f, State: Initialized, Role: FOLLOWER
I20260928 02:17:55.886520 9241 consensus_queue.cc:260] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a937508c9864d22b0a2b43949df43df" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44153 } } peers { permanent_uuid: "d2efe484a0434b7298bc751b0ae6239f" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43519 } } peers { permanent_uuid: "006c8083331e428c88ec278c62a12c44" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 46147 } }
I20260928 02:17:55.886641 9242 raft_consensus.cc:348] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a937508c9864d22b0a2b43949df43df" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44153 } } peers { permanent_uuid: "d2efe484a0434b7298bc751b0ae6239f" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43519 } } peers { permanent_uuid: "006c8083331e428c88ec278c62a12c44" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 46147 } }
I20260928 02:17:55.887187 9242 raft_consensus.cc:374] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:55.887390 9242 raft_consensus.cc:729] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 006c8083331e428c88ec278c62a12c44, State: Initialized, Role: FOLLOWER
I20260928 02:17:55.888037 9091 heartbeater.cc:503] Master 127.4.140.254:43149 was elected leader, sending a full tablet report...
I20260928 02:17:55.888157 9242 consensus_queue.cc:260] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a937508c9864d22b0a2b43949df43df" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44153 } } peers { permanent_uuid: "d2efe484a0434b7298bc751b0ae6239f" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43519 } } peers { permanent_uuid: "006c8083331e428c88ec278c62a12c44" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 46147 } }
I20260928 02:17:55.888732 9241 ts_tablet_manager.cc:1428] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260928 02:17:55.890687 9242 ts_tablet_manager.cc:1428] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260928 02:17:55.891881 9243 ts_tablet_manager.cc:1428] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260928 02:17:55.892232 9223 heartbeater.cc:503] Master 127.4.140.254:43149 was elected leader, sending a full tablet report...
I20260928 02:17:55.957216 9247 raft_consensus.cc:482] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:17:55.957588 9247 raft_consensus.cc:504] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a937508c9864d22b0a2b43949df43df" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44153 } } peers { permanent_uuid: "d2efe484a0434b7298bc751b0ae6239f" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43519 } } peers { permanent_uuid: "006c8083331e428c88ec278c62a12c44" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 46147 } }
I20260928 02:17:55.959492 9247 leader_election.cc:288] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2efe484a0434b7298bc751b0ae6239f (127.4.140.194:43519), 006c8083331e428c88ec278c62a12c44 (127.4.140.193:46147)
I20260928 02:17:55.966962 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:55.960100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:36809 (local address 127.4.140.193:46147)
0928 02:17:55.960322 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:55.960329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:55.960355 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:17:55.960526 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:55.960533 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:55.960623 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:55.960739 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:55.961184 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.962285 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.964704 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.966046 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.966234 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:55.966320 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:55.966574 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 02:17:55.966712 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:17:55.966993 9225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:55.959803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:46147 (local address 127.4.140.195:36809)
0928 02:17:55.960176 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:55.960198 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:55.960325 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:55.960773 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:55.960777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:55.960807 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:55.961030 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:55.961043 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.962419 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.962423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:55.964546 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:55.964560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.966225 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.966232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:55.966390 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:55.966414 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:17:55.966526 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 02:17:55.966673 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":76}
I20260928 02:17:55.967339 9240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:55.960051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:44815 (local address 127.4.140.194:43519)
0928 02:17:55.960239 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:55.960246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:55.960924 (+ 678us) server_negotiation.cc:407] Connection header received
0928 02:17:55.961124 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:55.961130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:55.961205 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:55.961322 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:17:55.961754 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.962658 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.964947 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.966400 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.966555 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:55.966626 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:55.966900 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 02:17:55.967047 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:17:55.967563 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:55.960074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:43519 (local address 127.4.140.195:44815)
0928 02:17:55.960808 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:55.960823 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:55.960936 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:55.961330 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:55.961333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:55.961351 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:17:55.961620 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:55.961626 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.962793 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.962798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:55.964806 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:55.964815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:55.966585 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:55.966590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:55.966740 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:55.966768 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 02:17:55.966861 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 02:17:55.966968 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":421,"threads_started":1}
I20260928 02:17:55.968143 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86b06fe19fb45099139c740946ddd75" candidate_uuid: "7a937508c9864d22b0a2b43949df43df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "006c8083331e428c88ec278c62a12c44" is_pre_election: true
I20260928 02:17:55.968652 9046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86b06fe19fb45099139c740946ddd75" candidate_uuid: "7a937508c9864d22b0a2b43949df43df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2efe484a0434b7298bc751b0ae6239f" is_pre_election: true
I20260928 02:17:55.968750 8913 raft_consensus.cc:2454] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a937508c9864d22b0a2b43949df43df in term 0.
I20260928 02:17:55.969097 9046 raft_consensus.cc:2454] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a937508c9864d22b0a2b43949df43df in term 0.
I20260928 02:17:55.969542 9114 leader_election.cc:302] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 006c8083331e428c88ec278c62a12c44, 7a937508c9864d22b0a2b43949df43df; no voters:
I20260928 02:17:55.969967 9247 raft_consensus.cc:2781] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:17:55.970086 9247 raft_consensus.cc:482] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:17:55.970224 9247 raft_consensus.cc:3037] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:55.972481 9247 raft_consensus.cc:504] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a937508c9864d22b0a2b43949df43df" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44153 } } peers { permanent_uuid: "d2efe484a0434b7298bc751b0ae6239f" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43519 } } peers { permanent_uuid: "006c8083331e428c88ec278c62a12c44" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 46147 } }
I20260928 02:17:55.973076 9247 leader_election.cc:288] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [CANDIDATE]: Term 1 election: Requested vote from peers d2efe484a0434b7298bc751b0ae6239f (127.4.140.194:43519), 006c8083331e428c88ec278c62a12c44 (127.4.140.193:46147)
I20260928 02:17:55.973651 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86b06fe19fb45099139c740946ddd75" candidate_uuid: "7a937508c9864d22b0a2b43949df43df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "006c8083331e428c88ec278c62a12c44"
I20260928 02:17:55.973675 9046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86b06fe19fb45099139c740946ddd75" candidate_uuid: "7a937508c9864d22b0a2b43949df43df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2efe484a0434b7298bc751b0ae6239f"
I20260928 02:17:55.973865 9046 raft_consensus.cc:3037] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:55.973865 8913 raft_consensus.cc:3037] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:55.977077 9046 raft_consensus.cc:2454] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a937508c9864d22b0a2b43949df43df in term 1.
I20260928 02:17:55.977077 8913 raft_consensus.cc:2454] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a937508c9864d22b0a2b43949df43df in term 1.
I20260928 02:17:55.977579 9112 leader_election.cc:302] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a937508c9864d22b0a2b43949df43df, d2efe484a0434b7298bc751b0ae6239f; no voters:
I20260928 02:17:55.977942 9247 raft_consensus.cc:2781] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:55.978427 9247 raft_consensus.cc:686] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [term 1 LEADER]: Becoming Leader. State: Replica: 7a937508c9864d22b0a2b43949df43df, State: Running, Role: LEADER
I20260928 02:17:55.979000 9247 consensus_queue.cc:237] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a937508c9864d22b0a2b43949df43df" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44153 } } peers { permanent_uuid: "d2efe484a0434b7298bc751b0ae6239f" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43519 } } peers { permanent_uuid: "006c8083331e428c88ec278c62a12c44" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 46147 } }
I20260928 02:17:55.983510 8768 catalog_manager.cc:5495] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a937508c9864d22b0a2b43949df43df (127.4.140.195). New cstate: current_term: 1 leader_uuid: "7a937508c9864d22b0a2b43949df43df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a937508c9864d22b0a2b43949df43df" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2efe484a0434b7298bc751b0ae6239f" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "006c8083331e428c88ec278c62a12c44" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 46147 } health_report { overall_health: UNKNOWN } } }
W20260928 02:17:55.989490 8959 tablet.cc:2369] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:17:55.998301 9224 tablet.cc:2369] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:17:56.002424 9092 tablet.cc:2369] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:17:56.026935 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:56.021073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.4.140.195:44153)
0928 02:17:56.021296 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:56.021302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:56.021486 (+ 184us) server_negotiation.cc:407] Connection header received
0928 02:17:56.021736 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:56.021741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:56.021847 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:56.022015 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:17:56.022459 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:56.023225 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:56.024561 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:56.024992 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:56.025104 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:56.025555 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:17:56.026138 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:17:56.026259 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:56.026520 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 02:17:56.026608 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:17:56.035307 8913 raft_consensus.cc:1261] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44 [term 1 FOLLOWER]: Refusing update from remote peer 7a937508c9864d22b0a2b43949df43df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:17:56.035321 9046 raft_consensus.cc:1261] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f [term 1 FOLLOWER]: Refusing update from remote peer 7a937508c9864d22b0a2b43949df43df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:17:56.036445 9251 consensus_queue.cc:1035] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2efe484a0434b7298bc751b0ae6239f" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 43519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:17:56.036806 9247 consensus_queue.cc:1035] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [LEADER]: Connected to new peer: Peer: permanent_uuid: "006c8083331e428c88ec278c62a12c44" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 46147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:17:56.049575 9256 mvcc.cc:205] Tried to move back new op lower bound from 7334141444230881280 to 7334141444013907968. Current Snapshot: MvccSnapshot[applied={T|T < 7334141444230881280}]
I20260928 02:17:56.062076 8770 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:17:56.062387 8770 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:17:56.066231 8770 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bb9d6bd76cc40819323259c7e942c68: Sending DeleteTablet for 3 replicas of tablet d86b06fe19fb45099139c740946ddd75
I20260928 02:17:56.067422 9158 tablet_service.cc:1514] Processing DeleteTablet for tablet d86b06fe19fb45099139c740946ddd75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:56 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:43290
I20260928 02:17:56.067462 8893 tablet_service.cc:1514] Processing DeleteTablet for tablet d86b06fe19fb45099139c740946ddd75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:56 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:37912
I20260928 02:17:56.068389 9026 tablet_service.cc:1514] Processing DeleteTablet for tablet d86b06fe19fb45099139c740946ddd75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:56 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:46402
I20260928 02:17:56.068974 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 8831
I20260928 02:17:56.071710 9263 tablet_replica.cc:330] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df: stopping tablet replica
I20260928 02:17:56.072331 9263 raft_consensus.cc:2229] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:17:56.073170 9263 raft_consensus.cc:2258] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:56.074012 9264 tablet_replica.cc:330] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44: stopping tablet replica
I20260928 02:17:56.074470 9264 raft_consensus.cc:2229] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:17:56.075011 9264 raft_consensus.cc:2258] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:56.075417 9263 ts_tablet_manager.cc:1905] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:56.077193 9264 ts_tablet_manager.cc:1905] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:56.081251 9265 tablet_replica.cc:330] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f: stopping tablet replica
I20260928 02:17:56.082297 9265 raft_consensus.cc:2229] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:17:56.082477 9263 ts_tablet_manager.cc:1918] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:17:56.082643 9263 log.cc:1196] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/d86b06fe19fb45099139c740946ddd75
I20260928 02:17:56.082993 9265 raft_consensus.cc:2258] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:56.083258 9263 ts_tablet_manager.cc:1939] T d86b06fe19fb45099139c740946ddd75 P 7a937508c9864d22b0a2b43949df43df: Deleting consensus metadata
I20260928 02:17:56.084604 9264 ts_tablet_manager.cc:1918] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:17:56.084908 9264 log.cc:1196] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/d86b06fe19fb45099139c740946ddd75
I20260928 02:17:56.086120 9264 ts_tablet_manager.cc:1939] T d86b06fe19fb45099139c740946ddd75 P 006c8083331e428c88ec278c62a12c44: Deleting consensus metadata
I20260928 02:17:56.086233 8755 catalog_manager.cc:4841] TS 7a937508c9864d22b0a2b43949df43df (127.4.140.195:44153): tablet d86b06fe19fb45099139c740946ddd75 (table test-table [id=19e35ef318a44f349ed67efbfbf416ae]) successfully deleted
I20260928 02:17:56.086324 9265 ts_tablet_manager.cc:1905] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:56.087863 8757 catalog_manager.cc:4841] TS 006c8083331e428c88ec278c62a12c44 (127.4.140.193:46147): tablet d86b06fe19fb45099139c740946ddd75 (table test-table [id=19e35ef318a44f349ed67efbfbf416ae]) successfully deleted
Sep 28 02:17:56 dist-test-slave-h37r krb5kdc[8726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:17:56.095172 9265 ts_tablet_manager.cc:1918] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:17:56.095381 9265 log.cc:1196] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/d86b06fe19fb45099139c740946ddd75
I20260928 02:17:56.095875 9265 ts_tablet_manager.cc:1939] T d86b06fe19fb45099139c740946ddd75 P d2efe484a0434b7298bc751b0ae6239f: Deleting consensus metadata
I20260928 02:17:56.097935 8755 catalog_manager.cc:4841] TS d2efe484a0434b7298bc751b0ae6239f (127.4.140.194:43519): tablet d86b06fe19fb45099139c740946ddd75 (table test-table [id=19e35ef318a44f349ed67efbfbf416ae]) successfully deleted
I20260928 02:17:56.102144 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:56.070221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37924 (local address 127.4.140.193:46147)
0928 02:17:56.070440 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:56.070449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:56.081936 (+ 11487us) server_negotiation.cc:407] Connection header received
0928 02:17:56.082238 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:56.082244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:56.082331 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:56.082423 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:56.084562 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:56.085716 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:56.088523 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:56.089154 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:56.089254 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:56.094767 (+ 5513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:56.094786 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:56.094808 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:56.094867 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:56.098850 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:56.099779 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:56.099790 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:56.099794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:56.100008 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:56.100438 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:56.100443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:56.100447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:56.100909 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:56.101216 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:56.101512 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 02:17:56.101736 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:17:56.102851 8953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:56.213126 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 8831
I20260928 02:17:56.230643 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 8964
Sep 28 02:17:56 dist-test-slave-h37r krb5kdc[8726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.194@KRBTEST.COM
I20260928 02:17:56.247215 9240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:56.231670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46414 (local address 127.4.140.194:43519)
0928 02:17:56.231884 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:56.231915 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:56.231937 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:17:56.232007 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:56.232011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:56.232072 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:56.232187 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:56.233362 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:56.234130 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:56.235361 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:56.235853 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:56.236100 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:56.240811 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:56.240827 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:56.240853 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:56.240897 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:56.244124 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:56.244895 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:56.244905 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:56.244907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:56.245041 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:56.245472 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:56.245482 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:56.245484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:56.245937 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:56.246254 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:56.246586 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 02:17:56.246822 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:17:56.248170 9086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:56.338637 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 8964
I20260928 02:17:56.356551 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 9096
Sep 28 02:17:56 dist-test-slave-h37r krb5kdc[8726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.195@KRBTEST.COM
I20260928 02:17:56.373342 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:56.357320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43302 (local address 127.4.140.195:44153)
0928 02:17:56.357500 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:56.357506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:56.357608 (+ 102us) server_negotiation.cc:407] Connection header received
0928 02:17:56.357823 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:56.357828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:56.357919 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:56.358030 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:56.359249 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:56.360187 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:56.361417 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:56.361887 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:56.362053 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:56.366856 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:56.366868 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:56.366889 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:17:56.366929 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:56.370380 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:56.371092 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:56.371100 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:56.371102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:56.371276 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:17:56.371700 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:17:56.371706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:17:56.371709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:17:56.372157 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:56.372502 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:56.372756 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 02:17:56.372970 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:17:56.374305 9218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:56.503677 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 9096
I20260928 02:17:56.522701 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 8739
I20260928 02:17:56.523984 8800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:56.631173 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 8739
2026-09-28T02:17:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4270 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T02:17:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:17:56Z Disabled control of system clock
I20260928 02:17:56.699059 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:45581
--webserver_interface=127.4.140.254
--webserver_port=0
--builtin_ntp_servers=127.4.140.212:40985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:45581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:17:57.039309 9274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:57.039630 9274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:57.039714 9274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:57.047312 9274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:17:57.047430 9274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:57.047511 9274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:57.047616 9274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:17:57.047675 9274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:17:57.057345 9274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:40985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:45581
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:45581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:57.058714 9274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:57.060636 9274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:57.072736 9280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:57.072777 9282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:57.072736 9279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:57.074134 9274 server_base.cc:1034] running on GCE node
I20260928 02:17:57.074971 9274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:57.076491 9274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:57.077688 9274 hybrid_clock.cc:648] HybridClock initialized: now 1790561877077640 us; error 44 us; skew 500 ppm
I20260928 02:17:57.078047 9274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:17:57.080372 9274 webserver.cc:460] Webserver started at http://127.4.140.254:39939/ using document root <none> and password file <none>
I20260928 02:17:57.081202 9274 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:57.081331 9274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:57.081674 9274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:57.084877 9274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "564914acce3d431e96642526b01bd8e2"
format_stamp: "Formatted at 2026-09-28 02:17:57 on dist-test-slave-h37r"
server_key: "4648706ec79877300d748741a3b047e5"
server_key_iv: "71cb40f0979226fe19316a1761a44817"
server_key_version: "encryptionkey@0"
I20260928 02:17:57.085685 9274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "564914acce3d431e96642526b01bd8e2"
format_stamp: "Formatted at 2026-09-28 02:17:57 on dist-test-slave-h37r"
server_key: "4648706ec79877300d748741a3b047e5"
server_key_iv: "71cb40f0979226fe19316a1761a44817"
server_key_version: "encryptionkey@0"
I20260928 02:17:57.090916 9274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 02:17:57.094534 9288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:57.096165 9274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:17:57.096362 9274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "564914acce3d431e96642526b01bd8e2"
format_stamp: "Formatted at 2026-09-28 02:17:57 on dist-test-slave-h37r"
server_key: "4648706ec79877300d748741a3b047e5"
server_key_iv: "71cb40f0979226fe19316a1761a44817"
server_key_version: "encryptionkey@0"
I20260928 02:17:57.096587 9274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:57.102325 9274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:57.103089 9274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:57.103384 9274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:57.122112 9274 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:45581
I20260928 02:17:57.122145 9339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:45581 every 8 connection(s)
I20260928 02:17:57.123530 9274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:17:57.127564 9340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:57.128463 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 9274
I20260928 02:17:57.128687 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:17:57.129492 4659 external_mini_cluster.cc:1428] Setting key 6c625a44edb25d1a275ead6b899a6dcf
I20260928 02:17:57.138221 9343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:57.131667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55600 (local address 127.4.140.254:45581)
0928 02:17:57.132929 (+ 1262us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:57.132941 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:57.132984 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:17:57.134632 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:57.134653 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:57.135109 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:57.135454 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:57.136540 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:57.136547 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:57.136560 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:17:57.136573 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:57.136635 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:17:57.136651 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:57.136815 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:57.136936 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 02:17:57.137213 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":447,"threads_started":1}
I20260928 02:17:57.142130 9340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2: Bootstrap starting.
I20260928 02:17:57.145726 9340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:57.147017 9340 log.cc:824] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:57.154762 9340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2: No bootstrap required, opened a new log
I20260928 02:17:57.161906 9340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564914acce3d431e96642526b01bd8e2" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45581 } }
I20260928 02:17:57.162271 9340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:57.162380 9340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 564914acce3d431e96642526b01bd8e2, State: Initialized, Role: FOLLOWER
I20260928 02:17:57.163018 9340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564914acce3d431e96642526b01bd8e2" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45581 } }
I20260928 02:17:57.163251 9340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:17:57.163424 9340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:17:57.163600 9340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:57.165802 9340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564914acce3d431e96642526b01bd8e2" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45581 } }
I20260928 02:17:57.166291 9340 leader_election.cc:302] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 564914acce3d431e96642526b01bd8e2; no voters:
I20260928 02:17:57.166940 9340 leader_election.cc:288] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:17:57.167145 9345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:57.168190 9345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2 [term 1 LEADER]: Becoming Leader. State: Replica: 564914acce3d431e96642526b01bd8e2, State: Running, Role: LEADER
I20260928 02:17:57.168690 9345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564914acce3d431e96642526b01bd8e2" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45581 } }
I20260928 02:17:57.169216 9340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:17:57.171726 9346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "564914acce3d431e96642526b01bd8e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564914acce3d431e96642526b01bd8e2" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45581 } } }
I20260928 02:17:57.172540 9346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2 [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:57.171814 9347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 564914acce3d431e96642526b01bd8e2. Latest consensus state: current_term: 1 leader_uuid: "564914acce3d431e96642526b01bd8e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564914acce3d431e96642526b01bd8e2" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45581 } } }
I20260928 02:17:57.172986 9347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2 [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:57.176911 9352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:17:57.182096 9352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:17:57.190481 9352 catalog_manager.cc:1347] Generated new cluster ID: f9381df34719412d842dd348f7c4e8b4
I20260928 02:17:57.190591 9352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:17:57.218274 9352 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:17:57.218772 9352 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:17:57.229861 9352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2: Generated new TSK 0
I20260928 02:17:57.231036 9352 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:17:57.245362 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:45581
--builtin_ntp_servers=127.4.140.212:40985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:17:57.576825 9364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:57.577245 9364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:57.577423 9364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:57.587007 9364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:57.587132 9364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:57.587311 9364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:17:57.597302 9364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:40985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:45581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:57.598680 9364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:57.600766 9364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:57.614254 9369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:57.614329 9370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:57.614473 9372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:57.615692 9364 server_base.cc:1034] running on GCE node
I20260928 02:17:57.616355 9364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:57.617352 9364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:57.618621 9364 hybrid_clock.cc:648] HybridClock initialized: now 1790561877618551 us; error 63 us; skew 500 ppm
I20260928 02:17:57.618980 9364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:17:57.621308 9364 webserver.cc:460] Webserver started at http://127.4.140.193:36655/ using document root <none> and password file <none>
I20260928 02:17:57.622023 9364 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:57.622154 9364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:57.622473 9364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:57.625463 9364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "bfee835ce0844e7391be86187525466d"
format_stamp: "Formatted at 2026-09-28 02:17:57 on dist-test-slave-h37r"
server_key: "84d72557ab543ae385abb8e90b1b36c0"
server_key_iv: "99d963171a7ad2fffe1b34dad180852a"
server_key_version: "encryptionkey@0"
I20260928 02:17:57.626338 9364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "bfee835ce0844e7391be86187525466d"
format_stamp: "Formatted at 2026-09-28 02:17:57 on dist-test-slave-h37r"
server_key: "84d72557ab543ae385abb8e90b1b36c0"
server_key_iv: "99d963171a7ad2fffe1b34dad180852a"
server_key_version: "encryptionkey@0"
I20260928 02:17:57.631582 9364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:17:57.635391 9378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:57.636816 9364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:17:57.637027 9364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "bfee835ce0844e7391be86187525466d"
format_stamp: "Formatted at 2026-09-28 02:17:57 on dist-test-slave-h37r"
server_key: "84d72557ab543ae385abb8e90b1b36c0"
server_key_iv: "99d963171a7ad2fffe1b34dad180852a"
server_key_version: "encryptionkey@0"
I20260928 02:17:57.637269 9364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:57.643244 9364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:57.644116 9364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:57.644384 9364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:57.645370 9364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:57.646757 9364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:57.646894 9364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:57.647020 9364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:57.647106 9364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:57.681500 9364 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:36399
I20260928 02:17:57.681553 9490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:36399 every 8 connection(s)
I20260928 02:17:57.683004 9364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:17:57.686677 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 9364
I20260928 02:17:57.686897 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:17:57.687681 4659 external_mini_cluster.cc:1428] Setting key aefd0f7d817e10c9af8192c321311cea
I20260928 02:17:57.690604 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:45581
--builtin_ntp_servers=127.4.140.212:40985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 02:17:57.691674 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:57.685551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:42251 (local address 127.4.140.254:45581)
0928 02:17:57.685990 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:57.685994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:57.687381 (+ 1387us) server_negotiation.cc:407] Connection header received
0928 02:17:57.689331 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:57.689337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:57.689403 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:57.689537 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:57.690927 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:57.690931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:57.690935 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:17:57.690950 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:57.690992 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:17:57.691004 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:57.691134 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:57.691321 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 02:17:57.691386 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":236,"threads_started":1}
I20260928 02:17:57.692656 9494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:57.686067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:45581 (local address 127.4.140.193:42251)
0928 02:17:57.687204 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:57.687250 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:57.689065 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:57.689774 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:57.689783 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:57.689876 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:57.690586 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:17:57.690589 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:57.690626 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:17:57.690709 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:57.691101 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:57.691112 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:57.691121 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:17:57.691413 (+ 292us) client_negotiation.cc:240] Negotiation successful
0928 02:17:57.691677 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":287,"threads_started":1}
I20260928 02:17:57.694250 9491 heartbeater.cc:347] Connected to a master server at 127.4.140.254:45581
I20260928 02:17:57.694576 9491 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:57.695303 9491 heartbeater.cc:511] Master 127.4.140.254:45581 requested a full tablet report, sending...
I20260928 02:17:57.697413 9305 ts_manager.cc:194] Registered new tserver with Master: bfee835ce0844e7391be86187525466d (127.4.140.193:36399)
W20260928 02:17:58.014551 9495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:58.014886 9495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:58.015058 9495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:58.022405 9495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:58.022547 9495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:58.022729 9495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:17:58.032032 9495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:40985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:45581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:58.033432 9495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:58.035293 9495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:58.048898 9501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:58.049000 9503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:58.048981 9500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:58.050112 9495 server_base.cc:1034] running on GCE node
I20260928 02:17:58.050743 9495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:58.051661 9495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:58.052950 9495 hybrid_clock.cc:648] HybridClock initialized: now 1790561878052872 us; error 60 us; skew 500 ppm
I20260928 02:17:58.053262 9495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:17:58.055660 9495 webserver.cc:460] Webserver started at http://127.4.140.194:41059/ using document root <none> and password file <none>
I20260928 02:17:58.056401 9495 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:58.056489 9495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:58.056792 9495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:58.059587 9495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "b916bf84db644fcab38ac5f2ad158c1a"
format_stamp: "Formatted at 2026-09-28 02:17:58 on dist-test-slave-h37r"
server_key: "ef5d29689947454568175092e56f2bf7"
server_key_iv: "9ca7f6fec82ff41271f87cd8a4048e30"
server_key_version: "encryptionkey@0"
I20260928 02:17:58.060388 9495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "b916bf84db644fcab38ac5f2ad158c1a"
format_stamp: "Formatted at 2026-09-28 02:17:58 on dist-test-slave-h37r"
server_key: "ef5d29689947454568175092e56f2bf7"
server_key_iv: "9ca7f6fec82ff41271f87cd8a4048e30"
server_key_version: "encryptionkey@0"
I20260928 02:17:58.065414 9495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:17:58.068899 9509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:58.070544 9495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:17:58.070719 9495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "b916bf84db644fcab38ac5f2ad158c1a"
format_stamp: "Formatted at 2026-09-28 02:17:58 on dist-test-slave-h37r"
server_key: "ef5d29689947454568175092e56f2bf7"
server_key_iv: "9ca7f6fec82ff41271f87cd8a4048e30"
server_key_version: "encryptionkey@0"
I20260928 02:17:58.070966 9495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:58.076841 9495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:58.077697 9495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:58.078003 9495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:58.079087 9495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:58.080546 9495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:58.080631 9495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:58.080785 9495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:58.080848 9495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:58.114567 9495 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:41457
I20260928 02:17:58.114645 9621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:41457 every 8 connection(s)
I20260928 02:17:58.116039 9495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:17:58.120755 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 9495
I20260928 02:17:58.120975 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:17:58.121515 4659 external_mini_cluster.cc:1428] Setting key c5770342b36d6f6f423d7ab8cf4501dd
I20260928 02:17:58.124779 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:45581
--builtin_ntp_servers=127.4.140.212:40985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 02:17:58.125483 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:58.118652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:59211 (local address 127.4.140.254:45581)
0928 02:17:58.118860 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:58.118878 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:58.120435 (+ 1557us) server_negotiation.cc:407] Connection header received
0928 02:17:58.122782 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:58.122787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:58.122891 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:58.123010 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:58.124575 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:58.124580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:58.124585 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:17:58.124605 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:58.124643 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:17:58.124658 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:58.124788 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:58.125053 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 02:17:58.125125 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:17:58.127269 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:58.119269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:45581 (local address 127.4.140.194:59211)
0928 02:17:58.120244 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:58.120281 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:58.122475 (+ 2194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:58.123243 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:58.123251 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:58.123390 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:58.124220 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:17:58.124223 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:58.124271 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:17:58.124381 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:58.124779 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:58.124790 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:58.124801 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 02:17:58.125769 (+ 968us) client_negotiation.cc:240] Negotiation successful
0928 02:17:58.126033 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":260,"threads_started":1}
I20260928 02:17:58.128727 9622 heartbeater.cc:347] Connected to a master server at 127.4.140.254:45581
I20260928 02:17:58.129060 9622 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:58.129763 9622 heartbeater.cc:511] Master 127.4.140.254:45581 requested a full tablet report, sending...
I20260928 02:17:58.131230 9305 ts_manager.cc:194] Registered new tserver with Master: b916bf84db644fcab38ac5f2ad158c1a (127.4.140.194:41457)
W20260928 02:17:58.450356 9625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:58.450656 9625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:58.450759 9625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:58.458254 9625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:58.458361 9625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:58.458468 9625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:17:58.467800 9625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:40985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:45581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:58.469071 9625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:58.471022 9625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:58.484596 9630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:58.484638 9631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:58.484807 9633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:58.485277 9625 server_base.cc:1034] running on GCE node
I20260928 02:17:58.485872 9625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:58.486851 9625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:58.488072 9625 hybrid_clock.cc:648] HybridClock initialized: now 1790561878488025 us; error 60 us; skew 500 ppm
I20260928 02:17:58.488394 9625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:17:58.490718 9625 webserver.cc:460] Webserver started at http://127.4.140.195:35143/ using document root <none> and password file <none>
I20260928 02:17:58.491400 9625 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:58.491529 9625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:58.491837 9625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:58.494601 9625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "2c71bbf3f0474ee886eeeafe5632d30a"
format_stamp: "Formatted at 2026-09-28 02:17:58 on dist-test-slave-h37r"
server_key: "49e9e2ef606311780405fcb12197c470"
server_key_iv: "16b23e0c7b0f70b5d001f2d6bd987cda"
server_key_version: "encryptionkey@0"
I20260928 02:17:58.495393 9625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "2c71bbf3f0474ee886eeeafe5632d30a"
format_stamp: "Formatted at 2026-09-28 02:17:58 on dist-test-slave-h37r"
server_key: "49e9e2ef606311780405fcb12197c470"
server_key_iv: "16b23e0c7b0f70b5d001f2d6bd987cda"
server_key_version: "encryptionkey@0"
I20260928 02:17:58.500607 9625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:17:58.504150 9639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:58.505551 9625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:17:58.505729 9625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "2c71bbf3f0474ee886eeeafe5632d30a"
format_stamp: "Formatted at 2026-09-28 02:17:58 on dist-test-slave-h37r"
server_key: "49e9e2ef606311780405fcb12197c470"
server_key_iv: "16b23e0c7b0f70b5d001f2d6bd987cda"
server_key_version: "encryptionkey@0"
I20260928 02:17:58.505955 9625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:58.511606 9625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:58.512518 9625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:58.512792 9625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:58.513712 9625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:17:58.515115 9625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:17:58.515234 9625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:58.515370 9625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:17:58.515456 9625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:58.549034 9625 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:45515
I20260928 02:17:58.549088 9751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:45515 every 8 connection(s)
I20260928 02:17:58.550395 9625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:17:58.555138 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 9625
I20260928 02:17:58.555334 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:17:58.555758 4659 external_mini_cluster.cc:1428] Setting key 63c3c8c54a493b522e2fd69b0bbdee5a
I20260928 02:17:58.558871 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:58.553104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:35177 (local address 127.4.140.254:45581)
0928 02:17:58.553289 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:58.553294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:58.554761 (+ 1467us) server_negotiation.cc:407] Connection header received
0928 02:17:58.556675 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:58.556680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:58.556762 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:58.556871 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:58.558154 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:58.558159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:58.558165 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:17:58.558190 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:58.558222 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:17:58.558233 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:58.558349 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:58.558545 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 02:17:58.558608 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:17:58.559787 9754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:58.553711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:45581 (local address 127.4.140.195:35177)
0928 02:17:58.554604 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:58.554640 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:58.556413 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:58.557108 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:58.557116 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:58.557210 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:58.557868 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:17:58.557870 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:58.557906 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:17:58.557993 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:58.558327 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:58.558336 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:58.558344 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:17:58.558626 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 02:17:58.558879 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":270,"threads_started":1}
I20260928 02:17:58.561074 9752 heartbeater.cc:347] Connected to a master server at 127.4.140.254:45581
I20260928 02:17:58.561318 9752 heartbeater.cc:464] Registering TS with master...
I20260928 02:17:58.561832 9752 heartbeater.cc:511] Master 127.4.140.254:45581 requested a full tablet report, sending...
I20260928 02:17:58.563124 9305 ts_manager.cc:194] Registered new tserver with Master: 2c71bbf3f0474ee886eeeafe5632d30a (127.4.140.195:45515)
I20260928 02:17:58.570845 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:17:58.577651 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:58.575402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55606 (local address 127.4.140.254:45581)
0928 02:17:58.575646 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:58.575651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:58.576143 (+ 492us) server_negotiation.cc:407] Connection header received
0928 02:17:58.576256 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:58.576259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:58.576310 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:58.576395 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:58.577163 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:58.577167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:58.577171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:17:58.577185 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:58.577214 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:17:58.577225 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:58.577327 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:58.577359 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 02:17:58.577399 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260928 02:17:58.581525 9305 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:17:58.588229 9305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:17:58.614220 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:58.600333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:45515 (local address 127.0.0.1:37080)
0928 02:17:58.607784 (+ 7451us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:58.608602 (+ 818us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:58.608770 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:58.611358 (+ 2588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:58.611365 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:58.611484 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:58.612338 (+ 854us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:17:58.612341 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:58.612373 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:17:58.612454 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:58.613405 (+ 951us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:58.613415 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:58.613435 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 02:17:58.613610 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 02:17:58.613694 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7168,"thread_start_us":9740,"threads_started":1}
I20260928 02:17:58.614612 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:58.600389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59482 (local address 127.4.140.193:36399)
0928 02:17:58.611364 (+ 10975us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:58.611371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:58.611405 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:17:58.611458 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:58.611463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:58.611665 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:58.611812 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:58.613696 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:58.613706 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:58.613717 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:17:58.613744 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:58.613797 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:17:58.613816 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:58.613931 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:58.614171 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 02:17:58.614266 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10837,"thread_start_us":10717,"threads_started":1}
I20260928 02:17:58.614730 9768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:58.600350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:36399 (local address 127.0.0.1:59482)
0928 02:17:58.609990 (+ 9640us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:58.610426 (+ 436us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:58.610566 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:58.612113 (+ 1547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:58.612117 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:58.612139 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:58.613484 (+ 1345us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:17:58.613487 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:58.613501 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:17:58.613534 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:58.613995 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:58.614001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:58.614006 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:17:58.614207 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 02:17:58.614257 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9462,"thread_start_us":10406,"threads_started":1}
I20260928 02:17:58.614835 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:58.607156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:41457 (local address 127.0.0.1:60994)
0928 02:17:58.611009 (+ 3853us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:58.611025 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:58.611115 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:58.611684 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:58.611688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:58.611720 (+ 32us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:58.613235 (+ 1515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:17:58.613241 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:58.613258 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:17:58.613301 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:58.614057 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:58.614067 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:58.614074 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:17:58.614229 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 02:17:58.614280 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3726,"thread_start_us":1664,"threads_started":1}
I20260928 02:17:58.615495 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:58.607753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.4.140.194:41457)
0928 02:17:58.609431 (+ 1678us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:58.609457 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:58.611223 (+ 1766us) server_negotiation.cc:407] Connection header received
0928 02:17:58.611296 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:58.611305 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:58.611508 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:58.611752 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:58.613711 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:58.613719 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:58.613734 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:17:58.613756 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:58.613803 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:17:58.613825 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:58.613996 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:58.615147 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0928 02:17:58.615239 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1453,"thread_start_us":1701,"threads_started":1}
I20260928 02:17:58.615506 9767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:58.600895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37080 (local address 127.4.140.195:45515)
0928 02:17:58.610682 (+ 9787us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:58.610702 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:58.610787 (+ 85us) server_negotiation.cc:407] Connection header received
0928 02:17:58.610868 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:58.610880 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:58.611142 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:58.611986 (+ 844us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:58.613140 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:58.613153 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:58.613171 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:17:58.613190 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:58.613248 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:17:58.613271 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:58.615031 (+ 1760us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:58.615114 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 02:17:58.615209 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9614,"thread_start_us":9520,"threads_started":1}
I20260928 02:17:58.619051 9687 tablet_service.cc:1467] Processing CreateTablet for tablet 55aaca4dda644e4bbdffd83378d9ac4c (DEFAULT_TABLE table=test-table [id=e2f85bf0f5fd4759aa90a16a2aee4b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:58.619074 9557 tablet_service.cc:1467] Processing CreateTablet for tablet 55aaca4dda644e4bbdffd83378d9ac4c (DEFAULT_TABLE table=test-table [id=e2f85bf0f5fd4759aa90a16a2aee4b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:58.619074 9426 tablet_service.cc:1467] Processing CreateTablet for tablet 55aaca4dda644e4bbdffd83378d9ac4c (DEFAULT_TABLE table=test-table [id=e2f85bf0f5fd4759aa90a16a2aee4b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:17:58.620357 9687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55aaca4dda644e4bbdffd83378d9ac4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:58.620393 9557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55aaca4dda644e4bbdffd83378d9ac4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:58.620393 9426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55aaca4dda644e4bbdffd83378d9ac4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:58.632107 9770 tablet_bootstrap.cc:492] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a: Bootstrap starting.
I20260928 02:17:58.634656 9772 tablet_bootstrap.cc:492] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a: Bootstrap starting.
I20260928 02:17:58.636755 9770 tablet_bootstrap.cc:654] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:58.637531 9771 tablet_bootstrap.cc:492] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d: Bootstrap starting.
I20260928 02:17:58.638204 9772 tablet_bootstrap.cc:654] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:58.638270 9770 log.cc:824] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:58.639410 9772 log.cc:824] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:58.640985 9771 tablet_bootstrap.cc:654] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:58.641656 9770 tablet_bootstrap.cc:492] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a: No bootstrap required, opened a new log
I20260928 02:17:58.641952 9770 ts_tablet_manager.cc:1397] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 02:17:58.642122 9771 log.cc:824] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:58.642457 9772 tablet_bootstrap.cc:492] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a: No bootstrap required, opened a new log
I20260928 02:17:58.642709 9772 ts_tablet_manager.cc:1397] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 02:17:58.646690 9771 tablet_bootstrap.cc:492] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d: No bootstrap required, opened a new log
I20260928 02:17:58.646960 9771 ts_tablet_manager.cc:1397] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 02:17:58.652698 9770 raft_consensus.cc:348] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee835ce0844e7391be86187525466d" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36399 } } peers { permanent_uuid: "2c71bbf3f0474ee886eeeafe5632d30a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45515 } } peers { permanent_uuid: "b916bf84db644fcab38ac5f2ad158c1a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41457 } }
I20260928 02:17:58.653192 9770 raft_consensus.cc:374] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:58.653339 9770 raft_consensus.cc:729] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c71bbf3f0474ee886eeeafe5632d30a, State: Initialized, Role: FOLLOWER
I20260928 02:17:58.653112 9772 raft_consensus.cc:348] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee835ce0844e7391be86187525466d" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36399 } } peers { permanent_uuid: "2c71bbf3f0474ee886eeeafe5632d30a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45515 } } peers { permanent_uuid: "b916bf84db644fcab38ac5f2ad158c1a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41457 } }
I20260928 02:17:58.653594 9772 raft_consensus.cc:374] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:58.653712 9772 raft_consensus.cc:729] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b916bf84db644fcab38ac5f2ad158c1a, State: Initialized, Role: FOLLOWER
I20260928 02:17:58.653995 9771 raft_consensus.cc:348] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee835ce0844e7391be86187525466d" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36399 } } peers { permanent_uuid: "2c71bbf3f0474ee886eeeafe5632d30a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45515 } } peers { permanent_uuid: "b916bf84db644fcab38ac5f2ad158c1a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41457 } }
I20260928 02:17:58.654134 9770 consensus_queue.cc:260] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee835ce0844e7391be86187525466d" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36399 } } peers { permanent_uuid: "2c71bbf3f0474ee886eeeafe5632d30a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45515 } } peers { permanent_uuid: "b916bf84db644fcab38ac5f2ad158c1a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41457 } }
I20260928 02:17:58.654395 9771 raft_consensus.cc:374] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:58.654264 9772 consensus_queue.cc:260] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee835ce0844e7391be86187525466d" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36399 } } peers { permanent_uuid: "2c71bbf3f0474ee886eeeafe5632d30a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45515 } } peers { permanent_uuid: "b916bf84db644fcab38ac5f2ad158c1a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41457 } }
I20260928 02:17:58.654544 9771 raft_consensus.cc:729] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfee835ce0844e7391be86187525466d, State: Initialized, Role: FOLLOWER
I20260928 02:17:58.655159 9771 consensus_queue.cc:260] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee835ce0844e7391be86187525466d" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36399 } } peers { permanent_uuid: "2c71bbf3f0474ee886eeeafe5632d30a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45515 } } peers { permanent_uuid: "b916bf84db644fcab38ac5f2ad158c1a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41457 } }
I20260928 02:17:58.656615 9491 heartbeater.cc:503] Master 127.4.140.254:45581 was elected leader, sending a full tablet report...
I20260928 02:17:58.657512 9771 ts_tablet_manager.cc:1428] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260928 02:17:58.660665 9752 heartbeater.cc:503] Master 127.4.140.254:45581 was elected leader, sending a full tablet report...
I20260928 02:17:58.661167 9622 heartbeater.cc:503] Master 127.4.140.254:45581 was elected leader, sending a full tablet report...
I20260928 02:17:58.662770 9772 ts_tablet_manager.cc:1428] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
I20260928 02:17:58.664386 9770 ts_tablet_manager.cc:1428] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a: Time spent starting tablet: real 0.022s user 0.014s sys 0.006s
W20260928 02:17:58.685343 9492 tablet.cc:2369] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:17:58.740569 9778 raft_consensus.cc:482] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:17:58.740885 9778 raft_consensus.cc:504] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee835ce0844e7391be86187525466d" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36399 } } peers { permanent_uuid: "2c71bbf3f0474ee886eeeafe5632d30a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45515 } } peers { permanent_uuid: "b916bf84db644fcab38ac5f2ad158c1a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41457 } }
I20260928 02:17:58.742503 9778 leader_election.cc:288] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c71bbf3f0474ee886eeeafe5632d30a (127.4.140.195:45515), b916bf84db644fcab38ac5f2ad158c1a (127.4.140.194:41457)
I20260928 02:17:58.745970 9767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:58.743051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:57487 (local address 127.4.140.195:45515)
0928 02:17:58.743235 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:58.743243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:58.743985 (+ 742us) server_negotiation.cc:407] Connection header received
0928 02:17:58.744167 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:58.744172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:58.744239 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:58.744340 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:58.745329 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:58.745335 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:58.745340 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:17:58.745356 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:58.745394 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:17:58.745405 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:58.745537 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:58.745650 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 02:17:58.745712 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:17:58.746073 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:58.742961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:53865 (local address 127.4.140.194:41457)
0928 02:17:58.743162 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:58.743170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:58.744178 (+ 1008us) server_negotiation.cc:407] Connection header received
0928 02:17:58.744289 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:58.744292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:58.744372 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:58.744473 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:58.745520 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:58.745525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:58.745529 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:17:58.745546 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:58.745580 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:17:58.745598 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:58.745715 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:58.745795 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 02:17:58.745845 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:17:58.746258 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:58.742977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:45515 (local address 127.4.140.193:57487)
0928 02:17:58.743867 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:58.743882 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:58.744017 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:58.744382 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:58.744387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:58.744413 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:58.745138 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:17:58.745141 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:58.745153 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:17:58.745188 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:58.745528 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:58.745533 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:58.745538 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:17:58.745692 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 02:17:58.745746 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":549,"threads_started":1}
I20260928 02:17:58.746317 9780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:58.743063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:41457 (local address 127.4.140.193:53865)
0928 02:17:58.744015 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:58.744033 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:58.744178 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:58.744520 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:58.744524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:58.744540 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:17:58.745340 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:17:58.745342 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:17:58.745353 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:17:58.745385 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:17:58.745697 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:17:58.745702 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:17:58.745707 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:17:58.745839 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 02:17:58.745878 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":478,"threads_started":1}
I20260928 02:17:58.747249 9577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55aaca4dda644e4bbdffd83378d9ac4c" candidate_uuid: "bfee835ce0844e7391be86187525466d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b916bf84db644fcab38ac5f2ad158c1a" is_pre_election: true
I20260928 02:17:58.747377 9707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55aaca4dda644e4bbdffd83378d9ac4c" candidate_uuid: "bfee835ce0844e7391be86187525466d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c71bbf3f0474ee886eeeafe5632d30a" is_pre_election: true
I20260928 02:17:58.747766 9577 raft_consensus.cc:2454] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfee835ce0844e7391be86187525466d in term 0.
I20260928 02:17:58.747877 9707 raft_consensus.cc:2454] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfee835ce0844e7391be86187525466d in term 0.
I20260928 02:17:58.748528 9382 leader_election.cc:302] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2c71bbf3f0474ee886eeeafe5632d30a, b916bf84db644fcab38ac5f2ad158c1a, bfee835ce0844e7391be86187525466d; no voters:
I20260928 02:17:58.748886 9778 raft_consensus.cc:2781] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:17:58.749028 9778 raft_consensus.cc:482] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:17:58.749154 9778 raft_consensus.cc:3037] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:58.751349 9778 raft_consensus.cc:504] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee835ce0844e7391be86187525466d" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36399 } } peers { permanent_uuid: "2c71bbf3f0474ee886eeeafe5632d30a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45515 } } peers { permanent_uuid: "b916bf84db644fcab38ac5f2ad158c1a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41457 } }
I20260928 02:17:58.752034 9778 leader_election.cc:288] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [CANDIDATE]: Term 1 election: Requested vote from peers 2c71bbf3f0474ee886eeeafe5632d30a (127.4.140.195:45515), b916bf84db644fcab38ac5f2ad158c1a (127.4.140.194:41457)
I20260928 02:17:58.752425 9707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55aaca4dda644e4bbdffd83378d9ac4c" candidate_uuid: "bfee835ce0844e7391be86187525466d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c71bbf3f0474ee886eeeafe5632d30a"
I20260928 02:17:58.752621 9707 raft_consensus.cc:3037] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:58.752570 9577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55aaca4dda644e4bbdffd83378d9ac4c" candidate_uuid: "bfee835ce0844e7391be86187525466d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b916bf84db644fcab38ac5f2ad158c1a"
I20260928 02:17:58.752818 9577 raft_consensus.cc:3037] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:58.754922 9707 raft_consensus.cc:2454] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfee835ce0844e7391be86187525466d in term 1.
I20260928 02:17:58.755192 9577 raft_consensus.cc:2454] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfee835ce0844e7391be86187525466d in term 1.
I20260928 02:17:58.755479 9379 leader_election.cc:302] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c71bbf3f0474ee886eeeafe5632d30a, bfee835ce0844e7391be86187525466d; no voters:
I20260928 02:17:58.755932 9778 raft_consensus.cc:2781] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:58.759922 9778 raft_consensus.cc:686] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [term 1 LEADER]: Becoming Leader. State: Replica: bfee835ce0844e7391be86187525466d, State: Running, Role: LEADER
I20260928 02:17:58.760565 9778 consensus_queue.cc:237] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee835ce0844e7391be86187525466d" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36399 } } peers { permanent_uuid: "2c71bbf3f0474ee886eeeafe5632d30a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45515 } } peers { permanent_uuid: "b916bf84db644fcab38ac5f2ad158c1a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41457 } }
I20260928 02:17:58.765959 9304 catalog_manager.cc:5495] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d reported cstate change: term changed from 0 to 1, leader changed from <none> to bfee835ce0844e7391be86187525466d (127.4.140.193). New cstate: current_term: 1 leader_uuid: "bfee835ce0844e7391be86187525466d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfee835ce0844e7391be86187525466d" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c71bbf3f0474ee886eeeafe5632d30a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b916bf84db644fcab38ac5f2ad158c1a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41457 } health_report { overall_health: UNKNOWN } } }
W20260928 02:17:58.802050 9753 tablet.cc:2369] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:17:58.807821 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:58.805830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.4.140.193:36399)
0928 02:17:58.806071 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:58.806075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:58.806094 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:17:58.806242 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:58.806246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:58.806327 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:58.806460 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:58.807214 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:58.807219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:58.807222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:17:58.807237 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:58.807267 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:17:58.807278 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:58.807367 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:58.807472 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 02:17:58.807542 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 02:17:58.816891 9707 raft_consensus.cc:1261] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a [term 1 FOLLOWER]: Refusing update from remote peer bfee835ce0844e7391be86187525466d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:17:58.816910 9577 raft_consensus.cc:1261] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a [term 1 FOLLOWER]: Refusing update from remote peer bfee835ce0844e7391be86187525466d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:17:58.817970 9781 consensus_queue.cc:1035] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b916bf84db644fcab38ac5f2ad158c1a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:17:58.818363 9778 consensus_queue.cc:1035] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c71bbf3f0474ee886eeeafe5632d30a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:17:58.831526 9784 mvcc.cc:205] Tried to move back new op lower bound from 7334141455622119424 to 7334141455408209920. Current Snapshot: MvccSnapshot[applied={T|T < 7334141455622119424}]
I20260928 02:17:58.831962 9786 mvcc.cc:205] Tried to move back new op lower bound from 7334141455622119424 to 7334141455408209920. Current Snapshot: MvccSnapshot[applied={T|T < 7334141455622119424}]
I20260928 02:17:58.852694 9303 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:17:58.853008 9303 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:17:58.856962 9303 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 564914acce3d431e96642526b01bd8e2: Sending DeleteTablet for 3 replicas of tablet 55aaca4dda644e4bbdffd83378d9ac4c
I20260928 02:17:58.858312 9426 tablet_service.cc:1514] Processing DeleteTablet for tablet 55aaca4dda644e4bbdffd83378d9ac4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:58 UTC) from {username='slave'} at 127.0.0.1:59482
I20260928 02:17:58.858286 9557 tablet_service.cc:1514] Processing DeleteTablet for tablet 55aaca4dda644e4bbdffd83378d9ac4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:58 UTC) from {username='slave'} at 127.0.0.1:60994
I20260928 02:17:58.858364 9687 tablet_service.cc:1514] Processing DeleteTablet for tablet 55aaca4dda644e4bbdffd83378d9ac4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:17:58 UTC) from {username='slave'} at 127.0.0.1:37080
I20260928 02:17:58.862767 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 9364
I20260928 02:17:58.869984 9795 tablet_replica.cc:330] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d: stopping tablet replica
I20260928 02:17:58.871534 9795 raft_consensus.cc:2229] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:17:58.873338 9795 raft_consensus.cc:2258] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:58.873924 9796 tablet_replica.cc:330] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a: stopping tablet replica
I20260928 02:17:58.874343 9794 tablet_replica.cc:330] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a: stopping tablet replica
I20260928 02:17:58.874671 9796 raft_consensus.cc:2229] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:17:58.874954 9794 raft_consensus.cc:2229] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:17:58.875380 9796 raft_consensus.cc:2258] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:58.876271 9794 raft_consensus.cc:2258] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:17:58.878631 9795 ts_tablet_manager.cc:1905] T 55aaca4dda644e4bbdffd83378d9ac4c P bfee835ce0844e7391be86187525466d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:58.878679 9796 ts_tablet_manager.cc:1905] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:58.881114 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:58.864557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.4.140.193:36399)
0928 02:17:58.865167 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:58.865177 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:58.877219 (+ 12042us) server_negotiation.cc:407] Connection header received
0928 02:17:58.877791 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:58.877798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:58.877866 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:58.877956 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:58.880273 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:58.880279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:58.880284 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:17:58.880310 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:58.880352 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:17:58.880369 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:58.880483 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:58.880719 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 02:17:58.880787 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435}
I20260928 02:17:58.882143 9486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:58.883582 9794 ts_tablet_manager.cc:1905] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:17:58.887358 9796 ts_tablet_manager.cc:1918] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:17:58.887565 9796 log.cc:1196] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/55aaca4dda644e4bbdffd83378d9ac4c
I20260928 02:17:58.888196 9796 ts_tablet_manager.cc:1939] T 55aaca4dda644e4bbdffd83378d9ac4c P 2c71bbf3f0474ee886eeeafe5632d30a: Deleting consensus metadata
I20260928 02:17:58.890414 9289 catalog_manager.cc:4841] TS 2c71bbf3f0474ee886eeeafe5632d30a (127.4.140.195:45515): tablet 55aaca4dda644e4bbdffd83378d9ac4c (table test-table [id=e2f85bf0f5fd4759aa90a16a2aee4b31]) successfully deleted
I20260928 02:17:58.891031 9794 ts_tablet_manager.cc:1918] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:17:58.891244 9794 log.cc:1196] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/55aaca4dda644e4bbdffd83378d9ac4c
I20260928 02:17:58.891755 9794 ts_tablet_manager.cc:1939] T 55aaca4dda644e4bbdffd83378d9ac4c P b916bf84db644fcab38ac5f2ad158c1a: Deleting consensus metadata
I20260928 02:17:58.893486 9292 catalog_manager.cc:4841] TS b916bf84db644fcab38ac5f2ad158c1a (127.4.140.194:41457): tablet 55aaca4dda644e4bbdffd83378d9ac4c (table test-table [id=e2f85bf0f5fd4759aa90a16a2aee4b31]) successfully deleted
I20260928 02:17:58.977890 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 9364
I20260928 02:17:58.995698 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 9495
W20260928 02:17:58.995730 9291 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.140.193:36399 (error 108)
I20260928 02:17:58.997793 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:58.997303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:36399 (local address 127.0.0.1:59526)
0928 02:17:58.997486 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:58.997627 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:36399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 02:17:58.998020 9291 catalog_manager.cc:4584] TS bfee835ce0844e7391be86187525466d (127.4.140.193:36399): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 55aaca4dda644e4bbdffd83378d9ac4c: Network error: Client connection negotiation failed: client connection to 127.4.140.193:36399: connect: Connection refused (error 111)
I20260928 02:17:58.999202 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:58.996849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32770 (local address 127.4.140.194:41457)
0928 02:17:58.997018 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:58.997022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:58.997038 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:17:58.997114 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:58.997118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:58.997229 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:58.997335 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:58.998523 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:58.998527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:58.998531 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:17:58.998546 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:58.998576 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:17:58.998587 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:58.998692 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:58.998845 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 02:17:58.998889 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:17:59.000200 9617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:59.051838 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:59.051256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:36399 (local address 127.0.0.1:59538)
0928 02:17:59.051509 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:59.051637 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:36399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 02:17:59.053046 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:59.052603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:36399 (local address 127.0.0.1:59540)
0928 02:17:59.052784 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:59.052872 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:36399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 02:17:59.089587 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 9495
I20260928 02:17:59.106452 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 9625
I20260928 02:17:59.109586 9767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:59.107223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37090 (local address 127.4.140.195:45515)
0928 02:17:59.107468 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:59.107473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:59.107495 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:17:59.107555 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:59.107559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:59.107638 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:59.107759 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:59.108847 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:59.108852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:59.108855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:17:59.108871 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:59.108902 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:17:59.108914 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:59.109102 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:59.109173 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 02:17:59.109242 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:17:59.110319 9747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:59.113718 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:59.113137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:36399 (local address 127.0.0.1:59546)
0928 02:17:59.113405 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:59.113522 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:36399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260928 02:17:59.115161 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:59.114546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:36399 (local address 127.0.0.1:59556)
0928 02:17:59.114723 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:59.114836 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:36399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 02:17:59.195806 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:59.195233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:36399 (local address 127.0.0.1:59558)
0928 02:17:59.195498 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:59.195628 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:36399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260928 02:17:59.197250 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:59.196617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:36399 (local address 127.0.0.1:59570)
0928 02:17:59.196966 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:59.197073 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:36399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260928 02:17:59.200707 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 9625
I20260928 02:17:59.216864 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 9274
I20260928 02:17:59.218073 9335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:17:59.323619 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 9274
2026-09-28T02:17:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2692 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 02:17:59.366161 4659 mini_oidc.cc:152] Starting JWKS server
I20260928 02:17:59.368875 4659 webserver.cc:460] Webserver started at https://127.0.0.1:34901/ using document root <none> and password file <none>
I20260928 02:17:59.369192 4659 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:17:59.369879 4659 webserver.cc:460] Webserver started at http://127.0.0.1:41365/ using document root <none> and password file <none>
2026-09-28T02:17:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:17:59Z Disabled control of system clock
I20260928 02:17:59.397527 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:39309
--webserver_interface=127.4.140.254
--webserver_port=0
--builtin_ntp_servers=127.4.140.212:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:39309
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34901/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561835360118-4659-0/testchainca.cert with env {}
W20260928 02:17:59.720453 9808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:17:59.720763 9808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:17:59.720850 9808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:17:59.728209 9808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:17:59.728324 9808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:17:59.728363 9808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:17:59.728426 9808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:17:59.728469 9808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:17:59.728525 9808 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:17:59.728564 9808 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34901/jwks.json
I20260928 02:17:59.738930 9808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:39309
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:39309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34901/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561835360118-4659-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:17:59.740197 9808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:17:59.741501 9808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:17:59.753365 9813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:59.753389 9816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:17:59.753418 9814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:17:59.754202 9808 server_base.cc:1034] running on GCE node
I20260928 02:17:59.754930 9808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:17:59.756439 9808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:17:59.757640 9808 hybrid_clock.cc:648] HybridClock initialized: now 1790561879757598 us; error 35 us; skew 500 ppm
I20260928 02:17:59.758004 9808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:17:59.760442 9808 webserver.cc:460] Webserver started at http://127.4.140.254:41087/ using document root <none> and password file <none>
I20260928 02:17:59.761147 9808 fs_manager.cc:359] Metadata directory not provided
I20260928 02:17:59.761265 9808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:17:59.761580 9808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:17:59.764461 9808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "4d14ef19a230473a9f7009937ff9bd28"
format_stamp: "Formatted at 2026-09-28 02:17:59 on dist-test-slave-h37r"
server_key: "97f85e5a78086f33aca22f1d7519976c"
server_key_iv: "c10dab1c669531d5dc100a53067310d0"
server_key_version: "encryptionkey@0"
I20260928 02:17:59.765300 9808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "4d14ef19a230473a9f7009937ff9bd28"
format_stamp: "Formatted at 2026-09-28 02:17:59 on dist-test-slave-h37r"
server_key: "97f85e5a78086f33aca22f1d7519976c"
server_key_iv: "c10dab1c669531d5dc100a53067310d0"
server_key_version: "encryptionkey@0"
I20260928 02:17:59.770560 9808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:17:59.774057 9822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:17:59.775624 9808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:17:59.775867 9808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "4d14ef19a230473a9f7009937ff9bd28"
format_stamp: "Formatted at 2026-09-28 02:17:59 on dist-test-slave-h37r"
server_key: "97f85e5a78086f33aca22f1d7519976c"
server_key_iv: "c10dab1c669531d5dc100a53067310d0"
server_key_version: "encryptionkey@0"
I20260928 02:17:59.776177 9808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:17:59.802145 9808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:17:59.803131 9808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:17:59.803490 9808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:17:59.823503 9808 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:39309
I20260928 02:17:59.823608 9873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:39309 every 8 connection(s)
I20260928 02:17:59.825021 9808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:17:59.826939 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 9808
I20260928 02:17:59.827152 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:17:59.827834 4659 external_mini_cluster.cc:1428] Setting key bdd2747052224519868805375f33bd46
I20260928 02:17:59.829743 9874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:17:59.842574 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:59.829937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55148 (local address 127.4.140.254:39309)
0928 02:17:59.831001 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:59.831019 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:59.831060 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:17:59.832729 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:59.832751 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:59.833219 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:59.833582 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:17:59.834265 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:59.835708 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:59.837615 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:59.838350 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:59.838533 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:59.839841 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:17:59.839849 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:17:59.839867 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:17:59.840668 (+ 801us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:17:59.840756 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:17:59.840777 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:17:59.840888 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 02:17:59.841133 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 02:17:59.841555 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":353,"threads_started":1}
I20260928 02:17:59.843075 9874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28: Bootstrap starting.
I20260928 02:17:59.846809 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:17:59.846832 9874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28: Neither blocks nor log segments found. Creating new log.
I20260928 02:17:59.847800 9874 log.cc:824] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28: Log is configured to *not* fsync() on all Append() calls
I20260928 02:17:59.851006 9874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28: No bootstrap required, opened a new log
I20260928 02:17:59.858294 9874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d14ef19a230473a9f7009937ff9bd28" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 39309 } }
I20260928 02:17:59.858667 9874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:17:59.858774 9874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d14ef19a230473a9f7009937ff9bd28, State: Initialized, Role: FOLLOWER
I20260928 02:17:59.859323 9874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d14ef19a230473a9f7009937ff9bd28" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 39309 } }
I20260928 02:17:59.859555 9874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:17:59.859702 9874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:17:59.859872 9874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:17:59.862455 9874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d14ef19a230473a9f7009937ff9bd28" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 39309 } }
I20260928 02:17:59.862989 9874 leader_election.cc:302] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d14ef19a230473a9f7009937ff9bd28; no voters:
I20260928 02:17:59.863623 9874 leader_election.cc:288] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:17:59.863843 9879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:17:59.864658 9879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28 [term 1 LEADER]: Becoming Leader. State: Replica: 4d14ef19a230473a9f7009937ff9bd28, State: Running, Role: LEADER
I20260928 02:17:59.865199 9879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d14ef19a230473a9f7009937ff9bd28" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 39309 } }
I20260928 02:17:59.865823 9874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:17:59.868392 9881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d14ef19a230473a9f7009937ff9bd28. Latest consensus state: current_term: 1 leader_uuid: "4d14ef19a230473a9f7009937ff9bd28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d14ef19a230473a9f7009937ff9bd28" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 39309 } } }
I20260928 02:17:59.868419 9880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d14ef19a230473a9f7009937ff9bd28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d14ef19a230473a9f7009937ff9bd28" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 39309 } } }
I20260928 02:17:59.868955 9880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28 [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:59.868955 9881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28 [sys.catalog]: This master's current role is: LEADER
I20260928 02:17:59.869921 9884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:17:59.880450 9884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:17:59.890626 9884 catalog_manager.cc:1347] Generated new cluster ID: a897ef2a1f424c25ac59595c4ba30235
I20260928 02:17:59.890761 9884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:17:59.907426 9884 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:17:59.909013 9884 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:17:59.916536 9884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d14ef19a230473a9f7009937ff9bd28: Generated new TSK 0
I20260928 02:17:59.917552 9884 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:17:59.965776 9904 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 02:17:59.965973 9904 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:17:59.967146 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:17:59.946272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55160 (local address 127.4.140.254:39309)
0928 02:17:59.946484 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 02:17:59.946491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:17:59.947120 (+ 629us) server_negotiation.cc:407] Connection header received
0928 02:17:59.947474 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:17:59.947478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:17:59.947548 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:17:59.947705 (+ 157us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:17:59.948311 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:59.949063 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:59.950521 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:17:59.950990 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:17:59.951130 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:59.951660 (+ 530us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:17:59.966416 (+ 14756us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34901/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:17:59.966792 (+ 376us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55160: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34901/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 02:17:59.967280 9877 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55160: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34901/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 02:17:59.968715 9903 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:17:59.946180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:39309 (local address 127.0.0.1:55160)
0928 02:17:59.947008 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0928 02:17:59.947025 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:17:59.947230 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:17:59.947736 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:17:59.947741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:17:59.947755 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:17:59.948149 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:59.948159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:59.949224 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:59.949230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:59.950369 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:17:59.950381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:17:59.951173 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:17:59.951181 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:17:59.951327 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:17:59.951383 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:17:59.966713 (+ 15330us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34901/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:17:59.967099 (+ 386us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.140.254:39309: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34901/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":467,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.4.140.254:39309: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34901/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 02:17:59.970665 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 9808
I20260928 02:17:59.971863 9869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:00.069419 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 9808
2026-09-28T02:18:00Z chronyd exiting
I20260928 02:18:00.119864 4659 test_util.cc:175] -----------------------------------------------
I20260928 02:18:00.120088 4659 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561835360118-4659-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (758 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 02:18:00.123682 4659 mini_oidc.cc:152] Starting JWKS server
I20260928 02:18:00.125511 4659 webserver.cc:460] Webserver started at https://127.0.0.1:39743/ using document root <none> and password file <none>
I20260928 02:18:00.125764 4659 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:18:00.126396 4659 webserver.cc:460] Webserver started at http://127.0.0.1:38875/ using document root <none> and password file <none>
2026-09-28T02:18:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:18:00Z Disabled control of system clock
I20260928 02:18:00.153294 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:36535
--webserver_interface=127.4.140.254
--webserver_port=0
--builtin_ntp_servers=127.4.140.212:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:36535
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39743/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561835360118-4659-0/testchainca.cert with env {}
W20260928 02:18:00.477540 9911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:00.477843 9911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:00.477931 9911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:00.485241 9911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:18:00.485363 9911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:00.485400 9911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:00.485450 9911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:18:00.485486 9911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:18:00.485540 9911 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:18:00.485580 9911 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39743/jwks.json
I20260928 02:18:00.495590 9911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:36535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:36535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39743/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561835360118-4659-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:00.496968 9911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:00.498318 9911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:00.509974 9916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:00.510123 9917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:00.510221 9919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:00.511650 9911 server_base.cc:1034] running on GCE node
I20260928 02:18:00.512497 9911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:00.513932 9911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:00.515170 9911 hybrid_clock.cc:648] HybridClock initialized: now 1790561880515122 us; error 42 us; skew 500 ppm
I20260928 02:18:00.515537 9911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:18:00.523574 9911 webserver.cc:460] Webserver started at http://127.4.140.254:38593/ using document root <none> and password file <none>
I20260928 02:18:00.524400 9911 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:00.524546 9911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:00.524883 9911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:00.527851 9911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "aee41b4731b548578070d58f73f77ee7"
format_stamp: "Formatted at 2026-09-28 02:18:00 on dist-test-slave-h37r"
server_key: "a26fb2c330cf95b7fa9bb1dbfa1b8fae"
server_key_iv: "82fbd559c945bbe5567b303baa74da19"
server_key_version: "encryptionkey@0"
I20260928 02:18:00.528753 9911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "aee41b4731b548578070d58f73f77ee7"
format_stamp: "Formatted at 2026-09-28 02:18:00 on dist-test-slave-h37r"
server_key: "a26fb2c330cf95b7fa9bb1dbfa1b8fae"
server_key_iv: "82fbd559c945bbe5567b303baa74da19"
server_key_version: "encryptionkey@0"
I20260928 02:18:00.534065 9911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:18:00.537649 9925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:00.538998 9911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:18:00.539182 9911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "aee41b4731b548578070d58f73f77ee7"
format_stamp: "Formatted at 2026-09-28 02:18:00 on dist-test-slave-h37r"
server_key: "a26fb2c330cf95b7fa9bb1dbfa1b8fae"
server_key_iv: "82fbd559c945bbe5567b303baa74da19"
server_key_version: "encryptionkey@0"
I20260928 02:18:00.539404 9911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:00.557255 9911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:00.558099 9911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:00.558432 9911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:00.576841 9911 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:36535
I20260928 02:18:00.576874 9976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:36535 every 8 connection(s)
I20260928 02:18:00.578215 9911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:18:00.581981 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 9911
I20260928 02:18:00.582122 9977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:00.582206 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:18:00.583032 4659 external_mini_cluster.cc:1428] Setting key 884598e91ae5bf9dd0b19bf1d031a584
I20260928 02:18:00.594619 9977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7: Bootstrap starting.
I20260928 02:18:00.596917 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:00.584940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46984 (local address 127.4.140.254:36535)
0928 02:18:00.585950 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:00.585961 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:00.586003 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:18:00.588247 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:00.588270 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:00.588755 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:00.589150 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:00.589705 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:00.590960 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:00.592391 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:00.593095 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:00.593300 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:00.594474 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:00.594482 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:00.594496 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:18:00.595133 (+ 637us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:00.595210 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:18:00.595230 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:00.595358 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:00.595592 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 02:18:00.595940 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":346,"threads_started":1}
I20260928 02:18:00.598014 9977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:00.599071 9977 log.cc:824] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:00.600783 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:18:00.602072 9977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7: No bootstrap required, opened a new log
I20260928 02:18:00.609632 9977 raft_consensus.cc:348] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee41b4731b548578070d58f73f77ee7" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36535 } }
I20260928 02:18:00.610067 9977 raft_consensus.cc:374] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:00.610180 9977 raft_consensus.cc:729] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee41b4731b548578070d58f73f77ee7, State: Initialized, Role: FOLLOWER
I20260928 02:18:00.610720 9977 consensus_queue.cc:260] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee41b4731b548578070d58f73f77ee7" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36535 } }
I20260928 02:18:00.610961 9977 raft_consensus.cc:388] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:18:00.611100 9977 raft_consensus.cc:482] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:18:00.611267 9977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:00.613525 9977 raft_consensus.cc:504] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee41b4731b548578070d58f73f77ee7" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36535 } }
I20260928 02:18:00.614099 9977 leader_election.cc:302] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aee41b4731b548578070d58f73f77ee7; no voters:
I20260928 02:18:00.614714 9977 leader_election.cc:288] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:18:00.614845 9982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:18:00.615576 9982 raft_consensus.cc:686] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7 [term 1 LEADER]: Becoming Leader. State: Replica: aee41b4731b548578070d58f73f77ee7, State: Running, Role: LEADER
I20260928 02:18:00.616120 9982 consensus_queue.cc:237] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee41b4731b548578070d58f73f77ee7" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36535 } }
I20260928 02:18:00.616845 9977 sys_catalog.cc:564] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:18:00.619112 9983 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aee41b4731b548578070d58f73f77ee7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee41b4731b548578070d58f73f77ee7" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36535 } } }
I20260928 02:18:00.619462 9983 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:00.619853 9984 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aee41b4731b548578070d58f73f77ee7. Latest consensus state: current_term: 1 leader_uuid: "aee41b4731b548578070d58f73f77ee7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee41b4731b548578070d58f73f77ee7" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36535 } } }
I20260928 02:18:00.620157 9984 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:00.621186 9989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:18:00.627765 9989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:18:00.636323 9989 catalog_manager.cc:1347] Generated new cluster ID: fbfd2a405e8349b8a0b3fe8a95755604
I20260928 02:18:00.636445 9989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:18:00.647472 9989 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:18:00.648782 9989 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:18:00.654752 9989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aee41b4731b548578070d58f73f77ee7: Generated new TSK 0
I20260928 02:18:00.655680 9989 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:18:00.704062 10007 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 02:18:00.704188 10007 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:18:00.705410 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:00.676882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46996 (local address 127.4.140.254:36535)
0928 02:18:00.677093 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:00.677101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:00.677477 (+ 376us) server_negotiation.cc:407] Connection header received
0928 02:18:00.677740 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:00.677744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:00.677814 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:00.677961 (+ 147us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:18:00.678477 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:00.679223 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:00.681333 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:00.681853 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:00.682026 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:00.682583 (+ 557us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:18:00.704815 (+ 22232us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39743/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:18:00.705168 (+ 353us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46996: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39743/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 02:18:00.705577 9980 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46996: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39743/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 02:18:00.705816 10006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:00.676700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36535 (local address 127.0.0.1:46996)
0928 02:18:00.677354 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:00.677378 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:00.677540 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:00.677985 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:00.677990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:00.678007 (+ 17us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:18:00.678306 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:00.678317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:00.679372 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:00.679379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:00.681162 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:00.681179 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:00.682060 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:00.682067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:00.682281 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:00.682327 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:18:00.705100 (+ 22773us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39743/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:18:00.705397 (+ 297us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.140.254:36535: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39743/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":327,"threads_started":1}
I20260928 02:18:00.707731 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 9911
I20260928 02:18:00.709009 9972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:00.807497 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 9911
2026-09-28T02:18:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (738 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 02:18:00.862738 4659 mini_oidc.cc:152] Starting JWKS server
I20260928 02:18:00.864715 4659 webserver.cc:460] Webserver started at https://127.0.0.1:39985/ using document root <none> and password file <none>
I20260928 02:18:00.864953 4659 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:18:00.865622 4659 webserver.cc:460] Webserver started at http://127.0.0.1:32971/ using document root <none> and password file <none>
2026-09-28T02:18:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:18:00Z Disabled control of system clock
I20260928 02:18:00.893055 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:34953
--webserver_interface=127.4.140.254
--webserver_port=0
--builtin_ntp_servers=127.4.140.212:36059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:34953
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39985/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:18:01.234010 10014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:01.234457 10014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:01.234622 10014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:01.242206 10014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:18:01.242384 10014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:01.242457 10014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:01.242542 10014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:18:01.242607 10014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:18:01.242703 10014 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:18:01.242760 10014 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39985/jwks.json
I20260928 02:18:01.252228 10014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:34953
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:34953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39985/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:01.253654 10014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:01.255499 10014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:01.267848 10022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:01.267961 10020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:01.267937 10019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:01.269143 10014 server_base.cc:1034] running on GCE node
I20260928 02:18:01.270015 10014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:01.271636 10014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:01.272899 10014 hybrid_clock.cc:648] HybridClock initialized: now 1790561881272819 us; error 76 us; skew 500 ppm
I20260928 02:18:01.273285 10014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:18:01.275693 10014 webserver.cc:460] Webserver started at http://127.4.140.254:37303/ using document root <none> and password file <none>
I20260928 02:18:01.276473 10014 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:01.276608 10014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:01.277026 10014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:01.279963 10014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "2c4cf0e3d3cb4a31b17c8f51d3198b72"
format_stamp: "Formatted at 2026-09-28 02:18:01 on dist-test-slave-h37r"
server_key: "1e34779d042fee22b98f9c3dc7bbc92b"
server_key_iv: "2e673d397e0e1490e4dfe6c1c577155e"
server_key_version: "encryptionkey@0"
I20260928 02:18:01.280797 10014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "2c4cf0e3d3cb4a31b17c8f51d3198b72"
format_stamp: "Formatted at 2026-09-28 02:18:01 on dist-test-slave-h37r"
server_key: "1e34779d042fee22b98f9c3dc7bbc92b"
server_key_iv: "2e673d397e0e1490e4dfe6c1c577155e"
server_key_version: "encryptionkey@0"
I20260928 02:18:01.286000 10014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:18:01.289559 10028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:01.290966 10014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:18:01.291167 10014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "2c4cf0e3d3cb4a31b17c8f51d3198b72"
format_stamp: "Formatted at 2026-09-28 02:18:01 on dist-test-slave-h37r"
server_key: "1e34779d042fee22b98f9c3dc7bbc92b"
server_key_iv: "2e673d397e0e1490e4dfe6c1c577155e"
server_key_version: "encryptionkey@0"
I20260928 02:18:01.291394 10014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:01.309262 10014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:01.310166 10014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:01.310497 10014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:01.329537 10014 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:34953
I20260928 02:18:01.329563 10079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:34953 every 8 connection(s)
I20260928 02:18:01.330984 10014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:18:01.333048 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 10014
I20260928 02:18:01.333235 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:18:01.334048 4659 external_mini_cluster.cc:1428] Setting key 341e5db72e05c40893a5b617ed91e301
I20260928 02:18:01.336481 10080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:01.348443 10083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:01.335787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54774 (local address 127.4.140.254:34953)
0928 02:18:01.336860 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:01.336870 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:01.336912 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:18:01.338430 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:01.338451 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:01.338915 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:01.339266 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:01.339846 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:01.341256 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:01.343138 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:01.343805 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:01.344025 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:01.345226 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:01.345236 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:01.345255 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:18:01.346141 (+ 886us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:01.346241 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:18:01.346269 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:01.346380 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:01.346651 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 02:18:01.347026 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":369,"threads_started":1}
I20260928 02:18:01.350466 10080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72: Bootstrap starting.
I20260928 02:18:01.352723 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:18:01.354072 10080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:01.355270 10080 log.cc:824] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:01.358095 10080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72: No bootstrap required, opened a new log
I20260928 02:18:01.365931 10080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4cf0e3d3cb4a31b17c8f51d3198b72" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 34953 } }
I20260928 02:18:01.366330 10080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:01.366438 10080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4cf0e3d3cb4a31b17c8f51d3198b72, State: Initialized, Role: FOLLOWER
I20260928 02:18:01.367122 10080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4cf0e3d3cb4a31b17c8f51d3198b72" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 34953 } }
I20260928 02:18:01.367380 10080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:18:01.367532 10080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:18:01.367745 10080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:01.370010 10080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4cf0e3d3cb4a31b17c8f51d3198b72" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 34953 } }
I20260928 02:18:01.370483 10080 leader_election.cc:302] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c4cf0e3d3cb4a31b17c8f51d3198b72; no voters:
I20260928 02:18:01.371070 10080 leader_election.cc:288] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:18:01.371300 10085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:18:01.372116 10085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72 [term 1 LEADER]: Becoming Leader. State: Replica: 2c4cf0e3d3cb4a31b17c8f51d3198b72, State: Running, Role: LEADER
I20260928 02:18:01.372690 10085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4cf0e3d3cb4a31b17c8f51d3198b72" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 34953 } }
I20260928 02:18:01.373329 10080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:18:01.376255 10086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c4cf0e3d3cb4a31b17c8f51d3198b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4cf0e3d3cb4a31b17c8f51d3198b72" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 34953 } } }
I20260928 02:18:01.376241 10087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c4cf0e3d3cb4a31b17c8f51d3198b72. Latest consensus state: current_term: 1 leader_uuid: "2c4cf0e3d3cb4a31b17c8f51d3198b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4cf0e3d3cb4a31b17c8f51d3198b72" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 34953 } } }
I20260928 02:18:01.376677 10086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:01.376730 10087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:01.378278 10091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:18:01.383435 10091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:18:01.393707 10091 catalog_manager.cc:1347] Generated new cluster ID: 491124561d794e15929b0cef99bebbf9
I20260928 02:18:01.393848 10091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:18:01.408013 10091 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:18:01.410006 10091 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:18:01.420691 10091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c4cf0e3d3cb4a31b17c8f51d3198b72: Generated new TSK 0
I20260928 02:18:01.421779 10091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:18:01.449481 10083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:01.441682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.4.140.254:34953)
0928 02:18:01.442025 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:01.442034 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:01.442615 (+ 581us) server_negotiation.cc:407] Connection header received
0928 02:18:01.442891 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:01.442895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:01.442975 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:01.443151 (+ 176us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:18:01.443601 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:01.444412 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:01.445768 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:01.446233 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:01.446373 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:01.446867 (+ 494us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:18:01.448808 (+ 1941us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39985/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:18:01.449136 (+ 328us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54782: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39985/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 02:18:01.449704 10109 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:01.441551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:34953 (local address 127.0.0.1:54782)
0928 02:18:01.442498 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:01.442518 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:01.442698 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:01.443164 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:01.443169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:01.443181 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:18:01.443456 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:01.443466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:01.444566 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:01.444572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:01.445617 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:01.445627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:01.446428 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:01.446438 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:01.446573 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:01.446606 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:18:01.449019 (+ 2413us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39985/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:18:01.449254 (+ 235us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.140.254:34953: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39985/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":681,"threads_started":1}
W20260928 02:18:01.450100 10083 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54782: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39985/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260928 02:18:01.450456 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 10014
I20260928 02:18:01.451604 10075 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 02:18:01.461849 10110 webserver.cc:533] Webserver: sslize failed:
E20260928 02:18:01.461993 10110 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:18:01.543772 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 10014
2026-09-28T02:18:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (736 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T02:18:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:18:01Z Disabled control of system clock
I20260928 02:18:01.626178 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:45553
--webserver_interface=127.4.140.254
--webserver_port=0
--builtin_ntp_servers=127.4.140.212:46139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:45553
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561835360118-4659-0/testchainca.cert with env {}
W20260928 02:18:01.950713 10115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:01.951011 10115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:01.951097 10115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:01.958174 10115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:18:01.958287 10115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:01.958324 10115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:01.958381 10115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:18:01.958426 10115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:18:01.958482 10115 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:18:01.958521 10115 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 02:18:01.968315 10115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:46139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:45553
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:45553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561835360118-4659-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:01.969583 10115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:01.970814 10115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:01.983158 10121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:01.983196 10123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:01.983713 10115 server_base.cc:1034] running on GCE node
W20260928 02:18:01.984002 10120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:01.984923 10115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:01.986258 10115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:01.987479 10115 hybrid_clock.cc:648] HybridClock initialized: now 1790561881987410 us; error 38 us; skew 500 ppm
I20260928 02:18:01.987844 10115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:18:01.990279 10115 webserver.cc:460] Webserver started at http://127.4.140.254:42689/ using document root <none> and password file <none>
I20260928 02:18:01.991045 10115 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:01.991161 10115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:01.991497 10115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:01.994447 10115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "570b02d1d20b43ac89b9eff71933448f"
format_stamp: "Formatted at 2026-09-28 02:18:01 on dist-test-slave-h37r"
server_key: "87e844c27c9c664475970fb551fc86e5"
server_key_iv: "55a0ac6ee0a11322fc60afcf7815ec16"
server_key_version: "encryptionkey@0"
I20260928 02:18:01.995253 10115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "570b02d1d20b43ac89b9eff71933448f"
format_stamp: "Formatted at 2026-09-28 02:18:01 on dist-test-slave-h37r"
server_key: "87e844c27c9c664475970fb551fc86e5"
server_key_iv: "55a0ac6ee0a11322fc60afcf7815ec16"
server_key_version: "encryptionkey@0"
I20260928 02:18:02.000553 10115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 02:18:02.004660 10129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:02.006803 10115 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:18:02.006999 10115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "570b02d1d20b43ac89b9eff71933448f"
format_stamp: "Formatted at 2026-09-28 02:18:01 on dist-test-slave-h37r"
server_key: "87e844c27c9c664475970fb551fc86e5"
server_key_iv: "55a0ac6ee0a11322fc60afcf7815ec16"
server_key_version: "encryptionkey@0"
I20260928 02:18:02.007292 10115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:02.029323 10115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:02.030133 10115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:02.030365 10115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:02.048777 10115 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:45553
I20260928 02:18:02.048780 10180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:45553 every 8 connection(s)
I20260928 02:18:02.050180 10115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:18:02.054067 10181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:02.055557 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 10115
I20260928 02:18:02.055774 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:18:02.056497 4659 external_mini_cluster.cc:1428] Setting key adc26ee856b64c6e5fbd259f7bd6accf
I20260928 02:18:02.067458 10181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f: Bootstrap starting.
I20260928 02:18:02.071190 10181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:02.072088 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:02.058464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.4.140.254:45553)
0928 02:18:02.059604 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:02.059620 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:02.059672 (+ 52us) server_negotiation.cc:407] Connection header received
0928 02:18:02.062041 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:02.062070 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:02.062641 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:02.063085 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:02.063623 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:02.065692 (+ 2069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:02.067002 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:02.068036 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:02.068201 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:02.069313 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:02.069325 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:02.069345 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:18:02.070033 (+ 688us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:02.070145 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:18:02.070170 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:02.070282 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:02.070487 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 02:18:02.070785 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":354,"threads_started":1}
I20260928 02:18:02.072504 10181 log.cc:824] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:02.075593 10181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f: No bootstrap required, opened a new log
I20260928 02:18:02.076159 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:18:02.083245 10181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570b02d1d20b43ac89b9eff71933448f" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45553 } }
I20260928 02:18:02.083611 10181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:02.083732 10181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 570b02d1d20b43ac89b9eff71933448f, State: Initialized, Role: FOLLOWER
I20260928 02:18:02.084375 10181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570b02d1d20b43ac89b9eff71933448f" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45553 } }
I20260928 02:18:02.084601 10181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:18:02.084748 10181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:18:02.084919 10181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:02.087023 10181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570b02d1d20b43ac89b9eff71933448f" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45553 } }
I20260928 02:18:02.087529 10181 leader_election.cc:302] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 570b02d1d20b43ac89b9eff71933448f; no voters:
I20260928 02:18:02.088222 10181 leader_election.cc:288] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:18:02.088395 10186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:18:02.089445 10186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f [term 1 LEADER]: Becoming Leader. State: Replica: 570b02d1d20b43ac89b9eff71933448f, State: Running, Role: LEADER
I20260928 02:18:02.089946 10186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570b02d1d20b43ac89b9eff71933448f" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45553 } }
I20260928 02:18:02.090242 10181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:18:02.092693 10188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 570b02d1d20b43ac89b9eff71933448f. Latest consensus state: current_term: 1 leader_uuid: "570b02d1d20b43ac89b9eff71933448f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570b02d1d20b43ac89b9eff71933448f" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45553 } } }
I20260928 02:18:02.093137 10188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:02.094031 10195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:18:02.093073 10187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "570b02d1d20b43ac89b9eff71933448f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570b02d1d20b43ac89b9eff71933448f" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45553 } } }
I20260928 02:18:02.094244 10187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:02.100482 10195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:18:02.109741 10195 catalog_manager.cc:1347] Generated new cluster ID: 9debab3bef30450782f6729935e29408
I20260928 02:18:02.109853 10195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:18:02.132931 10195 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:18:02.134280 10195 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:18:02.146854 10195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 570b02d1d20b43ac89b9eff71933448f: Generated new TSK 0
I20260928 02:18:02.147766 10195 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:18:02.255368 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:02.162132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45622 (local address 127.4.140.254:45553)
0928 02:18:02.162317 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:02.162324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:02.162860 (+ 536us) server_negotiation.cc:407] Connection header received
0928 02:18:02.163073 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:02.163079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:02.163182 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:02.163338 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:18:02.163787 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:02.164967 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:02.166452 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:02.167133 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:02.167256 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:02.167692 (+ 436us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:18:02.254606 (+ 86914us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:18:02.255039 (+ 433us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45622: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 02:18:02.255520 10184 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45622: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 02:18:02.255528 10210 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:02.162105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:45553 (local address 127.0.0.1:45622)
0928 02:18:02.162761 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:02.162777 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:02.162904 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:02.163363 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:02.163366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:02.163378 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:18:02.163647 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:02.163655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:02.165163 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:02.165167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:02.166307 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:02.166323 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:02.167295 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:02.167299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:02.167411 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:02.167451 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:18:02.254932 (+ 87481us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:18:02.255195 (+ 263us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.140.254:45553: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":355,"threads_started":1}
I20260928 02:18:02.257668 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 10115
I20260928 02:18:02.258850 10176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:02.362636 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 10115
2026-09-28T02:18:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (798 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (419 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (414 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:18:03 dist-test-slave-h37r krb5kdc[10225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:18:03 dist-test-slave-h37r krb5kdc[10225](info): set up 2 sockets
Sep 28 02:18:03 dist-test-slave-h37r krb5kdc[10225](info): commencing operation
krb5kdc: starting...
W20260928 02:18:04.292759 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:18:04 dist-test-slave-h37r krb5kdc[10225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:18:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:18:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:18:04.505514 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:33511
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:35893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:33511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:04.829600 10238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:04.829880 10238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:04.829970 10238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:04.837090 10238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:18:04.837196 10238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:04.837248 10238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:04.837306 10238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:18:04.837349 10238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:18:04.847074 10238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:35893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:33511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:33511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:04.848349 10238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:04.849684 10238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:04.861778 10243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:04.861809 10244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:04.861809 10246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:04.862419 10238 server_base.cc:1034] running on GCE node
I20260928 02:18:04.863313 10238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:04.864816 10238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:04.866032 10238 hybrid_clock.cc:648] HybridClock initialized: now 1790561884865980 us; error 41 us; skew 500 ppm
I20260928 02:18:04.866398 10238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:04 dist-test-slave-h37r krb5kdc[10225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561884, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:04.869772 10238 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:18:04.871366 10238 webserver.cc:460] Webserver started at http://127.4.140.254:39845/ using document root <none> and password file <none>
I20260928 02:18:04.872189 10238 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:04.872329 10238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:04.872679 10238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:04.875792 10238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "fd9c816ee4784d0d925dc0f2aa59b932"
format_stamp: "Formatted at 2026-09-28 02:18:04 on dist-test-slave-h37r"
server_key: "fcec0224bf1eb4232bf9ed17229477ab"
server_key_iv: "82b4967250a0d9d9da018dd808a5251e"
server_key_version: "encryptionkey@0"
I20260928 02:18:04.876672 10238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "fd9c816ee4784d0d925dc0f2aa59b932"
format_stamp: "Formatted at 2026-09-28 02:18:04 on dist-test-slave-h37r"
server_key: "fcec0224bf1eb4232bf9ed17229477ab"
server_key_iv: "82b4967250a0d9d9da018dd808a5251e"
server_key_version: "encryptionkey@0"
I20260928 02:18:04.881861 10238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:18:04.885512 10253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:04.887039 10238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:18:04.887247 10238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "fd9c816ee4784d0d925dc0f2aa59b932"
format_stamp: "Formatted at 2026-09-28 02:18:04 on dist-test-slave-h37r"
server_key: "fcec0224bf1eb4232bf9ed17229477ab"
server_key_iv: "82b4967250a0d9d9da018dd808a5251e"
server_key_version: "encryptionkey@0"
I20260928 02:18:04.887461 10238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:04.905182 10238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:04.909534 10238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:04.909860 10238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:04.929167 10238 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:33511
I20260928 02:18:04.929190 10304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:33511 every 8 connection(s)
I20260928 02:18:04.930678 10238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:18:04.934597 10305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:04.935380 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 10238
I20260928 02:18:04.935607 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:18:04.936434 4659 external_mini_cluster.cc:1428] Setting key d6c6280e95349e0901d3c73d08be5d81
I20260928 02:18:04.948987 10305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932: Bootstrap starting.
Sep 28 02:18:04 dist-test-slave-h37r krb5kdc[10225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:18:04.952780 10305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:04.953890 10305 log.cc:824] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:04.957007 10305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932: No bootstrap required, opened a new log
I20260928 02:18:04.959967 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:04.938538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45498 (local address 127.4.140.254:33511)
0928 02:18:04.939330 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:04.939342 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:04.939389 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:18:04.941089 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:04.941116 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:04.941525 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:04.941879 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:04.943554 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:04.944914 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:04.946772 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:04.947441 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:04.947571 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:04.952478 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:04.952486 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:04.952504 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:04.952543 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:04.955773 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:04.956526 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:04.956537 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:04.956549 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:04.956723 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:04.957104 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:04.957112 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:04.957118 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:04.957670 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:04.957993 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:04.958423 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 02:18:04.958868 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":367,"threads_started":1}
I20260928 02:18:04.964948 10305 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9c816ee4784d0d925dc0f2aa59b932" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 33511 } }
I20260928 02:18:04.965378 10305 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:04.965492 10305 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd9c816ee4784d0d925dc0f2aa59b932, State: Initialized, Role: FOLLOWER
I20260928 02:18:04.966043 10305 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9c816ee4784d0d925dc0f2aa59b932" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 33511 } }
I20260928 02:18:04.966315 10305 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:18:04.966464 10305 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:18:04.966681 10305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:04.969043 10305 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9c816ee4784d0d925dc0f2aa59b932" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 33511 } }
I20260928 02:18:04.969547 10305 leader_election.cc:302] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd9c816ee4784d0d925dc0f2aa59b932; no voters:
I20260928 02:18:04.970093 10305 leader_election.cc:288] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:18:04.970271 10310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:18:04.970963 10310 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932 [term 1 LEADER]: Becoming Leader. State: Replica: fd9c816ee4784d0d925dc0f2aa59b932, State: Running, Role: LEADER
I20260928 02:18:04.971659 10310 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9c816ee4784d0d925dc0f2aa59b932" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 33511 } }
I20260928 02:18:04.972548 10305 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:18:04.974776 10312 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd9c816ee4784d0d925dc0f2aa59b932. Latest consensus state: current_term: 1 leader_uuid: "fd9c816ee4784d0d925dc0f2aa59b932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9c816ee4784d0d925dc0f2aa59b932" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 33511 } } }
I20260928 02:18:04.974767 10311 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd9c816ee4784d0d925dc0f2aa59b932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9c816ee4784d0d925dc0f2aa59b932" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 33511 } } }
I20260928 02:18:04.975214 10311 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:04.975214 10312 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:04.976229 10315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:18:04.982213 10315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:18:04.993388 10315 catalog_manager.cc:1347] Generated new cluster ID: a3a53bd2f5494c32829b0cd3dfc3cd63
I20260928 02:18:04.993525 10315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:18:05.014509 10315 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:18:05.016042 10315 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:18:05.022377 10315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd9c816ee4784d0d925dc0f2aa59b932: Generated new TSK 0
I20260928 02:18:05.023515 10315 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
I20260928 02:18:05.091728 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:33511
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:35893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:05.425171 10331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:05.425549 10331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:05.425724 10331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:05.432775 10331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:05.432922 10331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:05.433113 10331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:18:05.442580 10331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:35893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:33511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:05.443915 10331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:05.445344 10331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:05.459100 10339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:05.459169 10336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:05.459131 10337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:05.459800 10331 server_base.cc:1034] running on GCE node
I20260928 02:18:05.460450 10331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:05.461454 10331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:05.462692 10331 hybrid_clock.cc:648] HybridClock initialized: now 1790561885462623 us; error 82 us; skew 500 ppm
I20260928 02:18:05.463016 10331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:05 dist-test-slave-h37r krb5kdc[10225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561885, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:05.466095 10331 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:18:05.467621 10331 webserver.cc:460] Webserver started at http://127.4.140.193:43665/ using document root <none> and password file <none>
I20260928 02:18:05.468397 10331 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:05.468549 10331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:05.468859 10331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:05.471752 10331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "198aac5633104fe1815bd54d2279d0f7"
format_stamp: "Formatted at 2026-09-28 02:18:05 on dist-test-slave-h37r"
server_key: "49f4cb676b19d804725cb59b589164f1"
server_key_iv: "9d3e4d4e4c806b029f24ea39961dee0c"
server_key_version: "encryptionkey@0"
I20260928 02:18:05.472675 10331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "198aac5633104fe1815bd54d2279d0f7"
format_stamp: "Formatted at 2026-09-28 02:18:05 on dist-test-slave-h37r"
server_key: "49f4cb676b19d804725cb59b589164f1"
server_key_iv: "9d3e4d4e4c806b029f24ea39961dee0c"
server_key_version: "encryptionkey@0"
I20260928 02:18:05.478101 10331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:18:05.481590 10346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:05.482914 10331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:18:05.483098 10331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "198aac5633104fe1815bd54d2279d0f7"
format_stamp: "Formatted at 2026-09-28 02:18:05 on dist-test-slave-h37r"
server_key: "49f4cb676b19d804725cb59b589164f1"
server_key_iv: "9d3e4d4e4c806b029f24ea39961dee0c"
server_key_version: "encryptionkey@0"
I20260928 02:18:05.483307 10331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:05.495332 10331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:05.499377 10331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:05.499671 10331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:05.500692 10331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:05.502048 10331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:05.502149 10331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:05.502295 10331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:05.502367 10331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:05.537298 10331 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:38367
I20260928 02:18:05.537333 10458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:38367 every 8 connection(s)
I20260928 02:18:05.538669 10331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:18:05.545047 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 10331
I20260928 02:18:05.545259 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:18:05.545869 4659 external_mini_cluster.cc:1428] Setting key 63dee14d4133f22e58769fb172bb4edb
Sep 28 02:18:05 dist-test-slave-h37r krb5kdc[10225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561885, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.140.194@KRBTEST.COM; defaulting to no policy
I20260928 02:18:05.565438 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:05.541251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:40859 (local address 127.4.140.254:33511)
0928 02:18:05.541734 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:05.541738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:05.542583 (+ 845us) server_negotiation.cc:407] Connection header received
0928 02:18:05.544472 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:05.544478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:05.544573 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:05.544727 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:05.546400 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:05.547266 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:05.548899 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:05.549287 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:05.549422 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:05.559634 (+ 10212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:05.559639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:05.559642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:05.559674 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:05.562275 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:05.562992 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:05.563002 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:05.563004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:05.563121 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:05.563559 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:05.563564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:05.563567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:05.563844 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:05.564087 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:05.564881 (+ 794us) server_negotiation.cc:299] Negotiation successful
0928 02:18:05.565073 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":293,"threads_started":1}
I20260928 02:18:05.566298 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:05.541695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:33511 (local address 127.4.140.193:40859)
0928 02:18:05.542411 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:05.542450 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:05.544199 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:05.545012 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:05.545020 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:05.545614 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:05.546228 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:05.546246 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:05.547449 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:05.547453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:05.548745 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:05.548757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:05.549513 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:05.549521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:05.549750 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:05.556723 (+ 6973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:05.556759 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:05.559381 (+ 2622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:05.562421 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:05.562431 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:05.562447 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:05.562860 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:05.563245 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:05.563250 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:05.563253 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:05.563437 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:05.564098 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:05.564110 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:05.564538 (+ 428us) client_negotiation.cc:769] Sending connection context
0928 02:18:05.564765 (+ 227us) client_negotiation.cc:240] Negotiation successful
0928 02:18:05.565197 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":300,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.194@KRBTEST.COM" created.
I20260928 02:18:05.568041 10459 heartbeater.cc:347] Connected to a master server at 127.4.140.254:33511
I20260928 02:18:05.568359 10459 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:05.569126 10459 heartbeater.cc:511] Master 127.4.140.254:33511 requested a full tablet report, sending...
I20260928 02:18:05.571519 10270 ts_manager.cc:194] Registered new tserver with Master: 198aac5633104fe1815bd54d2279d0f7 (127.4.140.193:38367)
I20260928 02:18:05.573395 10270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:40859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
I20260928 02:18:05.596218 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:33511
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:35893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:05.918133 10465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:05.918498 10465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:05.918674 10465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:05.925911 10465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:05.926081 10465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:05.926272 10465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:18:05.936304 10465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:35893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:33511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:05.937608 10465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:05.938918 10465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:05.952070 10473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:05.952150 10471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:05.952150 10470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:05.953030 10465 server_base.cc:1034] running on GCE node
I20260928 02:18:05.953825 10465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:05.954828 10465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:05.956046 10465 hybrid_clock.cc:648] HybridClock initialized: now 1790561885955983 us; error 57 us; skew 500 ppm
I20260928 02:18:05.956380 10465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:05 dist-test-slave-h37r krb5kdc[10225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561885, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:05.959525 10465 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:18:05.961023 10465 webserver.cc:460] Webserver started at http://127.4.140.194:36679/ using document root <none> and password file <none>
I20260928 02:18:05.961723 10465 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:05.961833 10465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:05.962087 10465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:05.965076 10465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "e08dc8a96c504d6daa6b4470ba63e43c"
format_stamp: "Formatted at 2026-09-28 02:18:05 on dist-test-slave-h37r"
server_key: "aff50bc40bd39e88f57bf432c7392112"
server_key_iv: "5641775521ace025931d3193e23b7d3d"
server_key_version: "encryptionkey@0"
I20260928 02:18:05.965826 10465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "e08dc8a96c504d6daa6b4470ba63e43c"
format_stamp: "Formatted at 2026-09-28 02:18:05 on dist-test-slave-h37r"
server_key: "aff50bc40bd39e88f57bf432c7392112"
server_key_iv: "5641775521ace025931d3193e23b7d3d"
server_key_version: "encryptionkey@0"
I20260928 02:18:05.971138 10465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:18:05.974969 10480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:05.976501 10465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:18:05.976686 10465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "e08dc8a96c504d6daa6b4470ba63e43c"
format_stamp: "Formatted at 2026-09-28 02:18:05 on dist-test-slave-h37r"
server_key: "aff50bc40bd39e88f57bf432c7392112"
server_key_iv: "5641775521ace025931d3193e23b7d3d"
server_key_version: "encryptionkey@0"
I20260928 02:18:05.976903 10465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:06.010623 10465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:06.015147 10465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:06.015436 10465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:06.016489 10465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:06.017858 10465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:06.017930 10465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:06.018106 10465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:06.018158 10465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:06.053023 10465 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:46699
I20260928 02:18:06.053076 10592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:46699 every 8 connection(s)
I20260928 02:18:06.054499 10465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:18:06.055859 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 10465
I20260928 02:18:06.056111 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:18:06.056615 4659 external_mini_cluster.cc:1428] Setting key 85df21ee21f9b4a2df51de18ed130b38
Sep 28 02:18:06 dist-test-slave-h37r krb5kdc[10225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561885, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:18:06.078012 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:06.057356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:51239 (local address 127.4.140.254:33511)
0928 02:18:06.057538 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:06.057543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:06.059030 (+ 1487us) server_negotiation.cc:407] Connection header received
0928 02:18:06.060953 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:06.060959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:06.061046 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:06.061177 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:06.062744 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:06.063455 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:06.065677 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:06.066471 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:06.066629 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:06.070589 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:06.070597 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:06.070601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:06.070647 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:06.074428 (+ 3781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:06.075266 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:06.075273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:06.075279 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:06.075449 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:06.075945 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:06.075953 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:06.075958 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:06.076353 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:06.076511 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:06.077262 (+ 751us) server_negotiation.cc:299] Negotiation successful
0928 02:18:06.077513 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.4.140.195@KRBTEST.COM; defaulting to no policy
I20260928 02:18:06.078931 10595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:06.057951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:33511 (local address 127.4.140.194:51239)
0928 02:18:06.058876 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:06.058923 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:06.060668 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:06.061424 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:06.061434 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:06.061963 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:06.062557 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:06.062575 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:06.063623 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:06.063627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:06.065520 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:06.065532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:06.066729 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:06.066737 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:06.067072 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:06.067774 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:06.067802 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:06.070334 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:06.074630 (+ 4296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:06.074638 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:06.074654 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:06.075112 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:06.075591 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:06.075595 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:06.075598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:06.075790 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:06.076491 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:06.076497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:06.076858 (+ 361us) client_negotiation.cc:769] Sending connection context
0928 02:18:06.077105 (+ 247us) client_negotiation.cc:240] Negotiation successful
0928 02:18:06.077510 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":443,"threads_started":1}
I20260928 02:18:06.080649 10593 heartbeater.cc:347] Connected to a master server at 127.4.140.254:33511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.195@KRBTEST.COM" created.
I20260928 02:18:06.081187 10593 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:06.082285 10593 heartbeater.cc:511] Master 127.4.140.254:33511 requested a full tablet report, sending...
I20260928 02:18:06.083937 10270 ts_manager.cc:194] Registered new tserver with Master: e08dc8a96c504d6daa6b4470ba63e43c (127.4.140.194:46699)
I20260928 02:18:06.084755 10270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:51239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
I20260928 02:18:06.107560 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:33511
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:35893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:06.434804 10598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:06.435163 10598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:06.435334 10598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:06.442802 10598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:06.442972 10598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:06.443161 10598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:18:06.453159 10598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:35893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:33511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:06.454537 10598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:06.456005 10598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:06.469786 10603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:06.469787 10604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:06.469823 10606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:06.470633 10598 server_base.cc:1034] running on GCE node
I20260928 02:18:06.471235 10598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:06.472244 10598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:06.473476 10598 hybrid_clock.cc:648] HybridClock initialized: now 1790561886473424 us; error 62 us; skew 500 ppm
I20260928 02:18:06.473834 10598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:06 dist-test-slave-h37r krb5kdc[10225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561886, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:06.477088 10598 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:18:06.478646 10598 webserver.cc:460] Webserver started at http://127.4.140.195:41139/ using document root <none> and password file <none>
I20260928 02:18:06.479429 10598 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:06.479574 10598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:06.479889 10598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:06.483027 10598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "a68ef8eb1b8040a399a9153d14fab445"
format_stamp: "Formatted at 2026-09-28 02:18:06 on dist-test-slave-h37r"
server_key: "9a4370516840ac61fb34745cfc7fae96"
server_key_iv: "4bead6d5cfec0c988d80c08594a2a253"
server_key_version: "encryptionkey@0"
I20260928 02:18:06.483780 10598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "a68ef8eb1b8040a399a9153d14fab445"
format_stamp: "Formatted at 2026-09-28 02:18:06 on dist-test-slave-h37r"
server_key: "9a4370516840ac61fb34745cfc7fae96"
server_key_iv: "4bead6d5cfec0c988d80c08594a2a253"
server_key_version: "encryptionkey@0"
I20260928 02:18:06.489154 10598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 02:18:06.492672 10613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:06.494098 10598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 02:18:06.494279 10598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "a68ef8eb1b8040a399a9153d14fab445"
format_stamp: "Formatted at 2026-09-28 02:18:06 on dist-test-slave-h37r"
server_key: "9a4370516840ac61fb34745cfc7fae96"
server_key_iv: "4bead6d5cfec0c988d80c08594a2a253"
server_key_version: "encryptionkey@0"
I20260928 02:18:06.494462 10598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:06.519268 10598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:06.523578 10598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:06.523921 10598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:06.524919 10598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:06.526288 10598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:06.526361 10598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:06.526537 10598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:06.526589 10598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:06.560210 10598 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:33487
I20260928 02:18:06.560294 10725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:33487 every 8 connection(s)
I20260928 02:18:06.561445 10598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:18:06.568317 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 10598
I20260928 02:18:06.568526 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:18:06.569044 4659 external_mini_cluster.cc:1428] Setting key b0695a7b426a864bd11e5e76d65584bc
Sep 28 02:18:06 dist-test-slave-h37r krb5kdc[10225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561886, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:18:06.577397 10459 heartbeater.cc:503] Master 127.4.140.254:33511 was elected leader, sending a full tablet report...
I20260928 02:18:06.582096 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:06.564089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:38665 (local address 127.4.140.254:33511)
0928 02:18:06.564261 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:06.564266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:06.565363 (+ 1097us) server_negotiation.cc:407] Connection header received
0928 02:18:06.567149 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:06.567154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:06.567267 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:06.567378 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:06.569273 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:06.570193 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:06.572315 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:06.572704 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:06.572796 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:06.576133 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:06.576137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:06.576140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:06.576171 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:06.578905 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:06.579788 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:06.579792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:06.579794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:06.579972 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:06.580357 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:06.580360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:06.580362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:06.580620 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:06.580792 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:06.581475 (+ 683us) server_negotiation.cc:299] Negotiation successful
0928 02:18:06.581701 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:18:06.583191 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:06.564529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:33511 (local address 127.4.140.195:38665)
0928 02:18:06.565202 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:06.565242 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:06.566881 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:06.567603 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:06.567611 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:06.568263 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:06.569051 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:06.569075 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:06.570344 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:06.570349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:06.572170 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:06.572187 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:06.572882 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:06.572887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:06.573069 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:06.573775 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:06.573827 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:06.575885 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:06.579030 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:06.579037 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:06.579055 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:06.579667 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:06.580066 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:06.580070 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:06.580072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:06.580258 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:06.580779 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:06.580788 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:06.581123 (+ 335us) client_negotiation.cc:769] Sending connection context
0928 02:18:06.581391 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 02:18:06.581946 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":269,"threads_started":1}
I20260928 02:18:06.584693 10726 heartbeater.cc:347] Connected to a master server at 127.4.140.254:33511
I20260928 02:18:06.585000 10726 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:06.585794 10726 heartbeater.cc:511] Master 127.4.140.254:33511 requested a full tablet report, sending...
I20260928 02:18:06.587168 10269 ts_manager.cc:194] Registered new tserver with Master: a68ef8eb1b8040a399a9153d14fab445 (127.4.140.195:33487)
I20260928 02:18:06.588130 10269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:38665
I20260928 02:18:06.596679 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/test-admin.keytab.
Sep 28 02:18:06 dist-test-slave-h37r krb5kdc[10225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:06.624959 4659 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 02:18:06.629976 10733 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:06.627614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38404 (local address 127.4.140.193:38367)
0928 02:18:06.628302 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:06.628309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:06.628335 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:18:06.628395 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:06.628402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:06.628609 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:06.628741 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:06.629531 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38404: BlockingRecv error: recv got EOF from 127.0.0.1:38404 (error 108)
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":394,"threads_started":1}
W20260928 02:18:06.629997 4659 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:18:06 dist-test-slave-h37r krb5kdc[10225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:06.631819 4659 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:18:06 dist-test-slave-h37r krb5kdc[10225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:06.633183 4659 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:18:06 dist-test-slave-h37r krb5kdc[10225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:18:06.648077 10733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:06.634821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38408 (local address 127.4.140.193:38367)
0928 02:18:06.635004 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:06.635012 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:06.635161 (+ 149us) server_negotiation.cc:407] Connection header received
0928 02:18:06.635289 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:06.635293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:06.635346 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:06.635427 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:06.636393 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:06.637335 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:06.638545 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:06.639092 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:06.639199 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:06.642664 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:06.642676 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:06.642691 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:06.642730 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:06.645384 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:06.645981 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:06.645988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:06.645990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:06.646121 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:06.646465 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:06.646470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:06.646472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:06.646838 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:06.647078 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:06.647526 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 02:18:06.647752 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 02:18:06.653939 10733 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:06.652084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.4.140.193:38367)
0928 02:18:06.652270 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:06.652278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:06.652557 (+ 279us) server_negotiation.cc:407] Connection header received
0928 02:18:06.652769 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:06.652773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:06.652837 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:06.652936 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:06.653706 (+ 770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38414: BlockingRecv error: recv got EOF from 127.0.0.1:38414 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 02:18:06.654134 4659 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:18:06 dist-test-slave-h37r krb5kdc[10225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:06.655702 4659 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:18:06 dist-test-slave-h37r krb5kdc[10225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:06.657071 4659 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:18:06 dist-test-slave-h37r krb5kdc[10225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:18:06.670691 10733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:06.658293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38424 (local address 127.4.140.193:38367)
0928 02:18:06.658534 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:06.658540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:06.658720 (+ 180us) server_negotiation.cc:407] Connection header received
0928 02:18:06.658831 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:06.658835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:06.658888 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:06.658967 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:06.660109 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:06.660814 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:06.662032 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:06.662480 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:06.662572 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:06.665984 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:06.665989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:06.665992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:06.666022 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:06.668431 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:06.669057 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:06.669062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:06.669064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:06.669186 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:06.669617 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:06.669620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:06.669622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:06.669872 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:06.670054 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:06.670297 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 02:18:06.670451 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 02:18:06.675338 10733 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:06.673540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38432 (local address 127.4.140.193:38367)
0928 02:18:06.673710 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:06.673714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:06.674077 (+ 363us) server_negotiation.cc:407] Connection header received
0928 02:18:06.674194 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:06.674197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:06.674270 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:06.674350 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:06.675029 (+ 679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38432: BlockingRecv error: recv got EOF from 127.0.0.1:38432 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 02:18:06.675521 4659 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561835360118-4659-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:18:06 dist-test-slave-h37r krb5kdc[10225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:06.677268 4659 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:18:06 dist-test-slave-h37r krb5kdc[10225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:06.678673 4659 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:18:06 dist-test-slave-h37r krb5kdc[10225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:18:06.692724 10733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:06.680018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38440 (local address 127.4.140.193:38367)
0928 02:18:06.680182 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:06.680186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:06.680582 (+ 396us) server_negotiation.cc:407] Connection header received
0928 02:18:06.680720 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:06.680723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:06.680777 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:06.680854 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:06.681864 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:06.682560 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:06.683714 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:06.684161 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:06.684252 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:06.687650 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:06.687656 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:06.687659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:06.687690 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:06.690089 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:06.690729 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:06.690733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:06.690735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:06.690862 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:06.691191 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:06.691195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:06.691196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:06.691664 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:06.691883 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:06.692189 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 02:18:06.692376 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:18:06.694314 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 10331
I20260928 02:18:06.705826 10733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:06.694964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38446 (local address 127.4.140.193:38367)
0928 02:18:06.695125 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:06.695129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:06.695541 (+ 412us) server_negotiation.cc:407] Connection header received
0928 02:18:06.695656 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:06.695659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:06.695709 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:06.695783 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:06.696701 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:06.697345 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:06.698531 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:06.699027 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:06.699122 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:06.701058 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:06.701062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:06.701065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:06.701092 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:06.703470 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:06.704210 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:06.704214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:06.704216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:06.704341 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:06.704725 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:06.704728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:06.704730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:06.704964 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:06.705150 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:06.705406 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 02:18:06.705558 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:18:06.706393 10454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:06.793200 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 10331
I20260928 02:18:06.809322 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 10465
Sep 28 02:18:06 dist-test-slave-h37r krb5kdc[10225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.194@KRBTEST.COM
I20260928 02:18:06.823760 10746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:06.810163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.4.140.194:46699)
0928 02:18:06.810682 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:06.810689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:06.810715 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:18:06.810776 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:06.810782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:06.811000 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:06.811139 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:06.812043 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:06.813115 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:06.814285 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:06.814816 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:06.814914 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:06.818523 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:06.818534 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:06.818548 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:06.818578 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:06.821262 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:06.821902 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:06.821909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:06.821911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:06.822026 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:06.822363 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:06.822366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:06.822368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:06.822686 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:06.822902 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:06.823161 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 02:18:06.823424 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":266,"threads_started":1}
I20260928 02:18:06.825032 10588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:06.911561 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 10465
I20260928 02:18:06.927708 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 10598
Sep 28 02:18:06 dist-test-slave-h37r krb5kdc[10225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.195@KRBTEST.COM
I20260928 02:18:06.945901 10748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:06.928786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35804 (local address 127.4.140.195:33487)
0928 02:18:06.929431 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:06.929439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:06.929469 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:18:06.929533 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:06.929540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:06.929800 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:06.929999 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:06.931203 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:06.932485 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:06.934447 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:06.935003 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:06.935123 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:06.939751 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:06.939764 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:06.939779 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:06.939820 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:06.942810 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:06.943548 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:06.943558 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:06.943560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:06.943693 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:06.944244 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:06.944253 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:06.944257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:06.944635 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:06.944962 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:06.945297 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 02:18:06.945577 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":320,"threads_started":1}
I20260928 02:18:06.947225 10721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:07.034008 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 10598
I20260928 02:18:07.052172 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 10238
I20260928 02:18:07.053361 10300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:07.157379 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 10238
2026-09-28T02:18:07Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3966 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:18:07 dist-test-slave-h37r krb5kdc[10753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:18:07 dist-test-slave-h37r krb5kdc[10753](info): set up 2 sockets
Sep 28 02:18:07 dist-test-slave-h37r krb5kdc[10753](info): commencing operation
krb5kdc: starting...
W20260928 02:18:08.252771 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:18:08 dist-test-slave-h37r krb5kdc[10753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:18:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:18:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.254.keytab.
Entry for principal oryx/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.254.keytab.
I20260928 02:18:08.471866 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:36419
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.254.keytab
--principal=oryx/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:36419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:08.809351 10766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:08.809679 10766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:08.809882 10766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:08.817343 10766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:18:08.817487 10766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:08.817540 10766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:08.817606 10766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:18:08.817657 10766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:18:08.827855 10766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:36419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.254.keytab
--principal=oryx/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:36419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:08.829216 10766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:08.830610 10766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:08.843289 10771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:08.843271 10772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:08.843580 10774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:08.844306 10766 server_base.cc:1034] running on GCE node
I20260928 02:18:08.845103 10766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:08.846442 10766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:08.847651 10766 hybrid_clock.cc:648] HybridClock initialized: now 1790561888847585 us; error 55 us; skew 500 ppm
I20260928 02:18:08.848024 10766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:08 dist-test-slave-h37r krb5kdc[10753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561888, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:08.851028 10766 init.cc:377] Logged in from keytab as oryx/127.4.140.254@KRBTEST.COM (short username oryx)
I20260928 02:18:08.852515 10766 webserver.cc:460] Webserver started at http://127.4.140.254:46303/ using document root <none> and password file <none>
I20260928 02:18:08.853237 10766 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:08.853340 10766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:08.853570 10766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:08.856494 10766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "4858ab829e994bd4a9c26227f4e81197"
format_stamp: "Formatted at 2026-09-28 02:18:08 on dist-test-slave-h37r"
server_key: "830d4db2503d6c8f713f30f70a30e37e"
server_key_iv: "cc7b69d91e4f217caa3351a1720d6053"
server_key_version: "encryptionkey@0"
I20260928 02:18:08.857198 10766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "4858ab829e994bd4a9c26227f4e81197"
format_stamp: "Formatted at 2026-09-28 02:18:08 on dist-test-slave-h37r"
server_key: "830d4db2503d6c8f713f30f70a30e37e"
server_key_iv: "cc7b69d91e4f217caa3351a1720d6053"
server_key_version: "encryptionkey@0"
I20260928 02:18:08.862411 10766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:18:08.865952 10781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:08.867393 10766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:18:08.867595 10766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "4858ab829e994bd4a9c26227f4e81197"
format_stamp: "Formatted at 2026-09-28 02:18:08 on dist-test-slave-h37r"
server_key: "830d4db2503d6c8f713f30f70a30e37e"
server_key_iv: "cc7b69d91e4f217caa3351a1720d6053"
server_key_version: "encryptionkey@0"
I20260928 02:18:08.867799 10766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:08.897931 10766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:08.902278 10766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:08.902582 10766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:08.921254 10766 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:36419
I20260928 02:18:08.921295 10832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:36419 every 8 connection(s)
I20260928 02:18:08.922883 10766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:18:08.926676 10833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:08.932582 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 10766
I20260928 02:18:08.932840 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:18:08.933651 4659 external_mini_cluster.cc:1428] Setting key a92767987a1746a55b151add201ac954
I20260928 02:18:08.937970 10833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197: Bootstrap starting.
I20260928 02:18:08.941743 10833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:08.942911 10833 log.cc:824] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:08.946214 10833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197: No bootstrap required, opened a new log
Sep 28 02:18:08 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM
I20260928 02:18:08.954507 10833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } }
I20260928 02:18:08.954916 10833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:08.955108 10833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4858ab829e994bd4a9c26227f4e81197, State: Initialized, Role: FOLLOWER
I20260928 02:18:08.955873 10833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } }
I20260928 02:18:08.956194 10833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:18:08.956332 10833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:18:08.956512 10833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:08.957623 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:08.935818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46756 (local address 127.4.140.254:36419)
0928 02:18:08.936692 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:08.936704 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:08.936744 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:18:08.938433 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:08.938457 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:08.938865 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:08.939187 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:08.940997 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:08.942380 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:08.944106 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:08.944862 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:08.944995 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:08.949514 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:08.949522 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:08.949539 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:08.949604 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:08.952863 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:08.953628 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:08.953635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:08.953646 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:08.953840 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:08.954232 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:08.954241 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:08.954243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:08.954836 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:08.955175 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:08.955621 (+ 446us) server_negotiation.cc:299] Negotiation successful
0928 02:18:08.956105 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":408,"threads_started":1}
I20260928 02:18:08.959685 10833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } }
I20260928 02:18:08.960255 10833 leader_election.cc:302] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4858ab829e994bd4a9c26227f4e81197; no voters:
I20260928 02:18:08.961102 10833 leader_election.cc:288] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:18:08.961249 10838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:18:08.961987 10838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [term 1 LEADER]: Becoming Leader. State: Replica: 4858ab829e994bd4a9c26227f4e81197, State: Running, Role: LEADER
I20260928 02:18:08.962478 10838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } }
I20260928 02:18:08.963207 10833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:18:08.965608 10840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4858ab829e994bd4a9c26227f4e81197. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } }
I20260928 02:18:08.966017 10840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:08.966028 10839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } }
I20260928 02:18:08.966411 10839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:08.967474 10846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:18:08.973142 10846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:18:08.983114 10846 catalog_manager.cc:1347] Generated new cluster ID: dfd33f39829443b9ac381d5072a1c40f
I20260928 02:18:08.983235 10846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:18:08.998399 10846 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:18:08.999706 10846 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:18:09.014569 10846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197: Generated new TSK 0
I20260928 02:18:09.015677 10846 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.193.keytab.
Entry for principal oryx/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.193.keytab.
I20260928 02:18:09.072428 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:36419
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.193.keytab
--principal=oryx/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:09.402040 10859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:09.402346 10859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:09.402513 10859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:09.409906 10859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:09.410037 10859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:09.410238 10859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:18:09.420244 10859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.193.keytab
--principal=oryx/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:36419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:09.421643 10859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:09.423002 10859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:09.436834 10867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:09.436906 10864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:09.436867 10865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:09.437674 10859 server_base.cc:1034] running on GCE node
I20260928 02:18:09.438285 10859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:09.439342 10859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:09.440611 10859 hybrid_clock.cc:648] HybridClock initialized: now 1790561889440562 us; error 49 us; skew 500 ppm
I20260928 02:18:09.441008 10859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:09 dist-test-slave-h37r krb5kdc[10753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561889, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:09.444569 10859 init.cc:377] Logged in from keytab as oryx/127.4.140.193@KRBTEST.COM (short username oryx)
I20260928 02:18:09.446267 10859 webserver.cc:460] Webserver started at http://127.4.140.193:41515/ using document root <none> and password file <none>
I20260928 02:18:09.447069 10859 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:09.447260 10859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:09.447597 10859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:09.450775 10859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "1d4a4eb5099e4262a21d9704f43f0bf1"
format_stamp: "Formatted at 2026-09-28 02:18:09 on dist-test-slave-h37r"
server_key: "b012f95b49c9b0ab8548b4b06d1d61db"
server_key_iv: "fd71463e303cbe10050dc0e6505561b1"
server_key_version: "encryptionkey@0"
I20260928 02:18:09.451614 10859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "1d4a4eb5099e4262a21d9704f43f0bf1"
format_stamp: "Formatted at 2026-09-28 02:18:09 on dist-test-slave-h37r"
server_key: "b012f95b49c9b0ab8548b4b06d1d61db"
server_key_iv: "fd71463e303cbe10050dc0e6505561b1"
server_key_version: "encryptionkey@0"
I20260928 02:18:09.457517 10859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 02:18:09.461305 10874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:09.463018 10859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:18:09.463193 10859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "1d4a4eb5099e4262a21d9704f43f0bf1"
format_stamp: "Formatted at 2026-09-28 02:18:09 on dist-test-slave-h37r"
server_key: "b012f95b49c9b0ab8548b4b06d1d61db"
server_key_iv: "fd71463e303cbe10050dc0e6505561b1"
server_key_version: "encryptionkey@0"
I20260928 02:18:09.463389 10859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:09.479557 10859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:09.483865 10859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:09.484212 10859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:09.485154 10859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:09.486475 10859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:09.486546 10859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:09.486707 10859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:09.486765 10859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:09.519685 10859 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:35389
I20260928 02:18:09.519774 10986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:35389 every 8 connection(s)
I20260928 02:18:09.521096 10859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:18:09.525774 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 10859
I20260928 02:18:09.526044 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:18:09.526850 4659 external_mini_cluster.cc:1428] Setting key 9a38d37163e39a81af629e9a47374bf1
Sep 28 02:18:09 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561889, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM
I20260928 02:18:09.543495 10989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:09.523876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:60017 (local address 127.4.140.254:36419)
0928 02:18:09.524334 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:09.524338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:09.525455 (+ 1117us) server_negotiation.cc:407] Connection header received
0928 02:18:09.528130 (+ 2675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:09.528136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:09.528216 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:09.528339 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:09.530421 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:09.531174 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:09.533650 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:09.534028 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:09.534133 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:09.537946 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:09.537951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:09.537955 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:09.537986 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:09.540484 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:09.541239 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:09.541251 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:09.541254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:09.541362 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:09.541780 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:09.541785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:09.541788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:09.542053 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:09.542203 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:09.542908 (+ 705us) server_negotiation.cc:299] Negotiation successful
0928 02:18:09.543131 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":211,"threads_started":1}
I20260928 02:18:09.544565 10990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:09.524487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.193:60017)
0928 02:18:09.525231 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:09.525271 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:09.527774 (+ 2503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:09.528637 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:09.528646 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:09.529372 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:09.530219 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:09.530247 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:09.531339 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:09.531347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:09.533499 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:09.533518 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:09.534264 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:09.534273 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:09.534612 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:09.535354 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:09.535389 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:09.537723 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:09.540641 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:09.540651 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:09.540668 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:09.541111 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:09.541475 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:09.541480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:09.541483 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:09.541663 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:09.542177 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:09.542185 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:09.542534 (+ 349us) client_negotiation.cc:769] Sending connection context
0928 02:18:09.542801 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 02:18:09.543324 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":280,"threads_started":1}
I20260928 02:18:09.546204 10987 heartbeater.cc:347] Connected to a master server at 127.4.140.254:36419
I20260928 02:18:09.546550 10987 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:09.547497 10987 heartbeater.cc:511] Master 127.4.140.254:36419 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.4.140.194@KRBTEST.COM; defaulting to no policy
I20260928 02:18:09.549784 10798 ts_manager.cc:194] Registered new tserver with Master: 1d4a4eb5099e4262a21d9704f43f0bf1 (127.4.140.193:35389)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.140.194@KRBTEST.COM" created.
I20260928 02:18:09.551405 10798 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:60017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.194.keytab.
Entry for principal oryx/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.194.keytab.
I20260928 02:18:09.577409 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:36419
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.194.keytab
--principal=oryx/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:09.902518 10993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:09.902819 10993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:09.902978 10993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:09.910503 10993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:09.910624 10993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:09.910797 10993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:18:09.920591 10993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.194.keytab
--principal=oryx/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:36419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:09.921945 10993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:09.923233 10993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:09.936296 10998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:09.936318 10999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:09.936964 11001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:09.937282 10993 server_base.cc:1034] running on GCE node
I20260928 02:18:09.938063 10993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:09.939034 10993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:09.940258 10993 hybrid_clock.cc:648] HybridClock initialized: now 1790561889940202 us; error 44 us; skew 500 ppm
I20260928 02:18:09.940653 10993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:09 dist-test-slave-h37r krb5kdc[10753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561889, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:09.943976 10993 init.cc:377] Logged in from keytab as oryx/127.4.140.194@KRBTEST.COM (short username oryx)
I20260928 02:18:09.945786 10993 webserver.cc:460] Webserver started at http://127.4.140.194:34875/ using document root <none> and password file <none>
I20260928 02:18:09.946605 10993 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:09.946740 10993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:09.947044 10993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:09.950026 10993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "f1736eee0e104b78ad42edc586263f47"
format_stamp: "Formatted at 2026-09-28 02:18:09 on dist-test-slave-h37r"
server_key: "10afb852b7e678375928854286168f5c"
server_key_iv: "e013b143eb55c8c00cc31528ef9aa512"
server_key_version: "encryptionkey@0"
I20260928 02:18:09.950776 10993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "f1736eee0e104b78ad42edc586263f47"
format_stamp: "Formatted at 2026-09-28 02:18:09 on dist-test-slave-h37r"
server_key: "10afb852b7e678375928854286168f5c"
server_key_iv: "e013b143eb55c8c00cc31528ef9aa512"
server_key_version: "encryptionkey@0"
I20260928 02:18:09.956053 10993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:18:09.959491 11008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:09.961007 10993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:18:09.961205 10993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "f1736eee0e104b78ad42edc586263f47"
format_stamp: "Formatted at 2026-09-28 02:18:09 on dist-test-slave-h37r"
server_key: "10afb852b7e678375928854286168f5c"
server_key_iv: "e013b143eb55c8c00cc31528ef9aa512"
server_key_version: "encryptionkey@0"
I20260928 02:18:09.961423 10993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:09.988379 10993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:09.992920 10993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:09.993217 10993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:09.994221 10993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:09.995747 10993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:09.995843 10993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:09.996004 10993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:09.996073 10993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:10.029152 10993 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:45109
I20260928 02:18:10.029197 11120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:45109 every 8 connection(s)
I20260928 02:18:10.030435 10993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:18:10.038442 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 10993
I20260928 02:18:10.038686 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:18:10.039255 4659 external_mini_cluster.cc:1428] Setting key 3a8592789dcc521d7302af68ac3ca576
Sep 28 02:18:10 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561889, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM
I20260928 02:18:10.051230 10989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:10.033029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:58685 (local address 127.4.140.254:36419)
0928 02:18:10.033195 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:10.033199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:10.034323 (+ 1124us) server_negotiation.cc:407] Connection header received
0928 02:18:10.036165 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:10.036170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:10.036262 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:10.036369 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:10.037838 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:10.038552 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:10.040648 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:10.041297 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:10.041433 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:10.045662 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:10.045667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:10.045671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:10.045700 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:10.048285 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:10.049004 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:10.049008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:10.049011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:10.049129 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:10.049545 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:10.049550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:10.049557 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:10.049805 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:10.049953 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:10.050659 (+ 706us) server_negotiation.cc:299] Negotiation successful
0928 02:18:10.050890 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:18:10.052322 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:10.033479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.194:58685)
0928 02:18:10.034162 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:10.034199 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:10.035858 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:10.036611 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:10.036619 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:10.037123 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:10.037681 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:10.037698 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:10.038720 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:10.038728 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:10.040473 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:10.040487 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:10.041560 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:10.041568 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:10.041862 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:10.042633 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:10.042669 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:10.045437 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:10.048425 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:10.048435 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:10.048454 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:10.048875 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:10.049233 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:10.049238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:10.049242 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:10.049426 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:10.049933 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:10.049941 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:10.050290 (+ 349us) client_negotiation.cc:769] Sending connection context
0928 02:18:10.050561 (+ 271us) client_negotiation.cc:240] Negotiation successful
0928 02:18:10.051121 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":255,"threads_started":1}
I20260928 02:18:10.053841 11121 heartbeater.cc:347] Connected to a master server at 127.4.140.254:36419
I20260928 02:18:10.054153 11121 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:10.054841 11121 heartbeater.cc:511] Master 127.4.140.254:36419 requested a full tablet report, sending...
I20260928 02:18:10.056371 10798 ts_manager.cc:194] Registered new tserver with Master: f1736eee0e104b78ad42edc586263f47 (127.4.140.194:45109)
I20260928 02:18:10.057173 10798 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:58685
WARNING: no policy specified for oryx/127.4.140.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.140.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.195.keytab.
Entry for principal oryx/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.195.keytab.
I20260928 02:18:10.087965 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:36419
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.195.keytab
--principal=oryx/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:10.412562 11126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:10.412837 11126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:10.412978 11126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:10.420076 11126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:10.420179 11126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:10.420351 11126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:18:10.429994 11126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.195.keytab
--principal=oryx/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:36419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:10.431304 11126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:10.432680 11126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:10.445434 11131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:10.445461 11132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:10.445487 11134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:10.446247 11126 server_base.cc:1034] running on GCE node
I20260928 02:18:10.446830 11126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:10.447767 11126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:10.448978 11126 hybrid_clock.cc:648] HybridClock initialized: now 1790561890448936 us; error 66 us; skew 500 ppm
I20260928 02:18:10.449285 11126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:10 dist-test-slave-h37r krb5kdc[10753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561890, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:10.452340 11126 init.cc:377] Logged in from keytab as oryx/127.4.140.195@KRBTEST.COM (short username oryx)
I20260928 02:18:10.453820 11126 webserver.cc:460] Webserver started at http://127.4.140.195:46811/ using document root <none> and password file <none>
I20260928 02:18:10.454509 11126 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:10.454641 11126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:10.454923 11126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:10.457897 11126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "c40dc0d1319b43d496ca89db9ca683c4"
format_stamp: "Formatted at 2026-09-28 02:18:10 on dist-test-slave-h37r"
server_key: "bbdbb19c5f7784df5f4fdb4f59abcf45"
server_key_iv: "d54fce7993954ded5c477d1b2ea810cd"
server_key_version: "encryptionkey@0"
I20260928 02:18:10.458634 11126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "c40dc0d1319b43d496ca89db9ca683c4"
format_stamp: "Formatted at 2026-09-28 02:18:10 on dist-test-slave-h37r"
server_key: "bbdbb19c5f7784df5f4fdb4f59abcf45"
server_key_iv: "d54fce7993954ded5c477d1b2ea810cd"
server_key_version: "encryptionkey@0"
I20260928 02:18:10.463526 11126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:18:10.467077 11141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:10.468654 11126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:18:10.468814 11126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "c40dc0d1319b43d496ca89db9ca683c4"
format_stamp: "Formatted at 2026-09-28 02:18:10 on dist-test-slave-h37r"
server_key: "bbdbb19c5f7784df5f4fdb4f59abcf45"
server_key_iv: "d54fce7993954ded5c477d1b2ea810cd"
server_key_version: "encryptionkey@0"
I20260928 02:18:10.469017 11126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:10.488819 11126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:10.492686 11126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:10.492965 11126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:10.493902 11126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:10.495216 11126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:10.495313 11126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:10.495463 11126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:10.495532 11126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:10.527709 11126 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:46077
I20260928 02:18:10.527729 11253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:46077 every 8 connection(s)
I20260928 02:18:10.528931 11126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:18:10.538035 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 11126
I20260928 02:18:10.538264 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:18:10.538771 4659 external_mini_cluster.cc:1428] Setting key 91f19bb6755daef57565f1657381e56f
Sep 28 02:18:10 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561890, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM
I20260928 02:18:10.548413 10989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:10.531444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:45777 (local address 127.4.140.254:36419)
0928 02:18:10.531691 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:10.531699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:10.533052 (+ 1353us) server_negotiation.cc:407] Connection header received
0928 02:18:10.534762 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:10.534767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:10.534856 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:10.534962 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:10.536467 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:10.537254 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:10.538825 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:10.539469 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:10.539619 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:10.543146 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:10.543152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:10.543155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:10.543186 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:10.545550 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:10.546233 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:10.546238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:10.546247 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:10.546363 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:10.546777 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:10.546782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:10.546785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:10.547035 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:10.547185 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:10.547820 (+ 635us) server_negotiation.cc:299] Negotiation successful
0928 02:18:10.548048 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:18:10.549340 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:10.532163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.195:45777)
0928 02:18:10.532894 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:10.532929 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:10.534506 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:10.535169 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:10.535177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:10.535662 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:10.536264 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:10.536281 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:10.537380 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:10.537383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:10.538684 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:10.538694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:10.539710 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:10.539720 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:10.540044 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:10.540766 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:10.540796 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:10.542935 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:10.545688 (+ 2753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:10.545697 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:10.545713 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:10.546112 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:10.546482 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:10.546487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:10.546490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:10.546664 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:10.547161 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:10.547169 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:10.547484 (+ 315us) client_negotiation.cc:769] Sending connection context
0928 02:18:10.547736 (+ 252us) client_negotiation.cc:240] Negotiation successful
0928 02:18:10.548292 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":311,"threads_started":1}
I20260928 02:18:10.550668 11254 heartbeater.cc:347] Connected to a master server at 127.4.140.254:36419
I20260928 02:18:10.550964 11254 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:10.551687 11254 heartbeater.cc:511] Master 127.4.140.254:36419 requested a full tablet report, sending...
I20260928 02:18:10.553320 10798 ts_manager.cc:194] Registered new tserver with Master: c40dc0d1319b43d496ca89db9ca683c4 (127.4.140.195:46077)
I20260928 02:18:10.554024 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:18:10.554417 10798 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:45777
I20260928 02:18:10.554935 10987 heartbeater.cc:503] Master 127.4.140.254:36419 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.4.140.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.140.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.140.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.253.keytab.
Entry for principal oryx/127.4.140.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.253.keytab.
I20260928 02:18:10.600286 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.253:45809
--webserver_interface=127.4.140.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.253.keytab
--principal=oryx/127.4.140.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:36419,127.4.140.253:45809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:10.924512 11259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:10.924862 11259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:10.925010 11259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:10.932760 11259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:18:10.932896 11259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:10.932989 11259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:10.933092 11259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:18:10.933162 11259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:18:10.943156 11259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:36419,127.4.140.253:45809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.253.keytab
--principal=oryx/127.4.140.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.253:45809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.140.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
W20260928 02:18:10.943542 11259 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.140.254:36419,127.4.140.253:45809'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 02:18:10.944650 11259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:10.945927 11259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:10.957628 11264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:10.957631 11267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:10.957661 11265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:10.958336 11259 server_base.cc:1034] running on GCE node
I20260928 02:18:10.959087 11259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:10.960474 11259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:10.961711 11259 hybrid_clock.cc:648] HybridClock initialized: now 1790561890961656 us; error 42 us; skew 500 ppm
I20260928 02:18:10.962040 11259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:10 dist-test-slave-h37r krb5kdc[10753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561890, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:10.972012 11259 init.cc:377] Logged in from keytab as oryx/127.4.140.253@KRBTEST.COM (short username oryx)
I20260928 02:18:10.973862 11259 webserver.cc:460] Webserver started at http://127.4.140.253:35821/ using document root <none> and password file <none>
I20260928 02:18:10.974596 11259 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:10.974716 11259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:10.975028 11259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:10.978101 11259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/data/instance:
uuid: "1878aa81226947478f7f8470d8f0452f"
format_stamp: "Formatted at 2026-09-28 02:18:10 on dist-test-slave-h37r"
server_key: "8c2fe841670435c05964eafa354ce724"
server_key_iv: "300cb08b1e44aa76eda25db1e995b218"
server_key_version: "encryptionkey@0"
I20260928 02:18:10.978858 11259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/wal/instance:
uuid: "1878aa81226947478f7f8470d8f0452f"
format_stamp: "Formatted at 2026-09-28 02:18:10 on dist-test-slave-h37r"
server_key: "8c2fe841670435c05964eafa354ce724"
server_key_iv: "300cb08b1e44aa76eda25db1e995b218"
server_key_version: "encryptionkey@0"
I20260928 02:18:10.986390 11259 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260928 02:18:10.990768 11274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:10.992750 11259 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 02:18:10.993049 11259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/wal
uuid: "1878aa81226947478f7f8470d8f0452f"
format_stamp: "Formatted at 2026-09-28 02:18:10 on dist-test-slave-h37r"
server_key: "8c2fe841670435c05964eafa354ce724"
server_key_iv: "300cb08b1e44aa76eda25db1e995b218"
server_key_version: "encryptionkey@0"
I20260928 02:18:10.993351 11259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:11.014905 11259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:11.021059 11259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:11.021376 11259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:11.047273 11259 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.253:45809
I20260928 02:18:11.047312 11325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.253:45809 every 8 connection(s)
I20260928 02:18:11.048704 11259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/data/info.pb
I20260928 02:18:11.049647 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 11259
I20260928 02:18:11.053992 11326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:11.059741 11121 heartbeater.cc:503] Master 127.4.140.254:36419 was elected leader, sending a full tablet report...
I20260928 02:18:11.063139 11326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:18:11 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.140.253@KRBTEST.COM
Sep 28 02:18:11 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561890, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.253@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM
I20260928 02:18:11.075753 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.052201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.4.140.253:45809)
0928 02:18:11.053170 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:11.053186 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:11.053584 (+ 398us) server_negotiation.cc:407] Connection header received
0928 02:18:11.055406 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:11.055427 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:11.055799 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:11.056199 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:11.058090 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.059402 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.061318 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.062412 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.062609 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.067454 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:11.067463 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:11.067483 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:11.067516 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:11.071034 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.071734 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.071744 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.071757 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.071988 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.072426 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.072429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.072431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.073047 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:11.073364 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:11.073976 (+ 612us) server_negotiation.cc:299] Negotiation successful
0928 02:18:11.074429 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"spinlock_wait_cycles":18304,"thread_start_us":394,"threads_started":1}
I20260928 02:18:11.082232 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.065969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.4.140.254:36419)
0928 02:18:11.067611 (+ 1642us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:11.067615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:11.067632 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:18:11.067707 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:11.067712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:11.067805 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:11.067951 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:11.069555 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.070303 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.072134 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.072748 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.072878 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.076675 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:11.076681 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:11.076685 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:11.076718 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:11.079842 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.080582 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.080586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.080588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.080709 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.081022 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.081024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.081026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.081300 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:11.081416 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:11.081828 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 02:18:11.081974 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1539,"thread_start_us":1444,"threads_started":1}
I20260928 02:18:11.082258 11331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.065978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.0.0.1:46766)
0928 02:18:11.066859 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:11.066905 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:11.067099 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:11.068010 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:11.068020 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:11.068976 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:11.069357 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.069367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.070453 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.070462 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.071987 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.071998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.072944 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.072950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.073129 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.073861 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:11.073881 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:11.076469 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:11.080005 (+ 3536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.080015 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.080020 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.080464 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.080802 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.080805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.080808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.080932 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.081402 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:11.081408 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:11.081593 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 02:18:11.081745 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 02:18:11.081971 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":394,"threads_started":1}
I20260928 02:18:11.083738 11326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:18:11 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561890, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.253@KRBTEST.COM for oryx/127.4.140.253@KRBTEST.COM
I20260928 02:18:11.095451 11331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.084353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:45809 (local address 127.0.0.1:53262)
0928 02:18:11.084544 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:11.084559 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:11.084650 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:11.084914 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:11.084918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:11.085248 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:11.085376 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.085384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.086291 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.086305 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.087214 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.087224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.087840 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.087844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.088169 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.088774 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:11.088794 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:11.090772 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:11.093376 (+ 2604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.093380 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.093383 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.093795 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.094131 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.094135 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.094137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.094258 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.094682 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:11.094686 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:11.094807 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 02:18:11.094920 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 02:18:11.095177 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 02:18:11.095651 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.084532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53262 (local address 127.4.140.253:45809)
0928 02:18:11.084669 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:11.084672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:11.084687 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:18:11.084724 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:11.084727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:11.084777 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:11.084849 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:11.085488 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.086160 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.087330 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.087718 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.087806 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.090913 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:11.090918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:11.090920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:11.090946 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:11.093259 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.093899 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.093903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.093905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.094036 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.094344 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.094346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.094348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.094591 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:11.094692 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:11.094990 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 02:18:11.095312 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":50}
I20260928 02:18:11.100461 11326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f: Bootstrap starting.
I20260928 02:18:11.103578 11326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:11.104624 11326 log.cc:824] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:11.107106 11326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f: No bootstrap required, opened a new log
I20260928 02:18:11.113948 11326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } }
I20260928 02:18:11.114269 11326 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:11.114339 11326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1878aa81226947478f7f8470d8f0452f, State: Initialized, Role: FOLLOWER
I20260928 02:18:11.114840 11326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } }
I20260928 02:18:11.115630 11333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } } }
I20260928 02:18:11.115949 11333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:18:11.116677 11326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:18:11.125389 11344 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:18:11.125568 11344 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:18:11.127254 11259 master_runner.cc:428] Detected that this master 1878aa81226947478f7f8470d8f0452f is joining an existing cluster
I20260928 02:18:11.127362 11259 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.140.253:45809
I20260928 02:18:11.150743 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.131859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53264 (local address 127.4.140.253:45809)
0928 02:18:11.132043 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:11.132050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:11.133936 (+ 1886us) server_negotiation.cc:407] Connection header received
0928 02:18:11.134122 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:11.134130 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:11.134222 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:11.134332 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:11.136190 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.137760 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.139799 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.140553 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.140619 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.143090 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:11.143097 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:11.143102 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:11.143137 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:11.147388 (+ 4251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.148526 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.148531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.148534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.148665 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.149097 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.149102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.149105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.149497 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:11.149632 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:11.150104 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 02:18:11.150320 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:18:11.151006 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.131458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:45809 (local address 127.0.0.1:53264)
0928 02:18:11.133840 (+ 2382us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:11.133866 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:11.134008 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:11.134513 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:11.134521 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:11.135435 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:11.136002 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.136021 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.138028 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.138039 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.139655 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.139674 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.140943 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.140947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.141047 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.141698 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:11.141721 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:11.142872 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:11.147526 (+ 4654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.147531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.147535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.148399 (+ 864us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.148812 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.148816 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.148820 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.148983 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.149698 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:11.149702 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:11.149846 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 02:18:11.150262 (+ 416us) client_negotiation.cc:240] Negotiation successful
0928 02:18:11.150512 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2247,"thread_start_us":2099,"threads_started":1}
I20260928 02:18:11.154189 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.131446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46768 (local address 127.4.140.254:36419)
0928 02:18:11.131648 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:11.131657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:11.138657 (+ 7000us) server_negotiation.cc:407] Connection header received
0928 02:18:11.138807 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:11.138813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:11.138905 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:11.139012 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:11.140409 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.141541 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.143459 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.144032 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.144163 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.146184 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:11.146190 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:11.146194 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:11.146232 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:11.149748 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.151639 (+ 1891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.151647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.151652 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.151810 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.152320 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.152328 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.152334 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.152698 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:11.152826 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:11.153485 (+ 659us) server_negotiation.cc:299] Negotiation successful
0928 02:18:11.153735 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:18:11.155424 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.132255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.0.0.1:46768)
0928 02:18:11.138532 (+ 6277us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:11.138554 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:11.138684 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:11.139191 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:11.139196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:11.139933 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:11.140241 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.140252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.141720 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.141726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.143319 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.143329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.144221 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.144228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.144404 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.144959 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:11.144975 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:11.145974 (+ 999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:11.149895 (+ 3921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.149899 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.149903 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.151497 (+ 1594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.151988 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.151995 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.152002 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.152189 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.152921 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:11.152931 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:11.153193 (+ 262us) client_negotiation.cc:769] Sending connection context
0928 02:18:11.154447 (+ 1254us) client_negotiation.cc:240] Negotiation successful
0928 02:18:11.154827 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6120,"spinlock_wait_cycles":1920,"thread_start_us":5890,"threads_started":1}
WARNING: no policy specified for oryx/127.4.140.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.140.252@KRBTEST.COM" created.
Sep 28 02:18:11 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561888, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.254@KRBTEST.COM for oryx/127.4.140.253@KRBTEST.COM
I20260928 02:18:11.181466 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.162259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53278 (local address 127.4.140.253:45809)
0928 02:18:11.162453 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:11.162464 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:11.163109 (+ 645us) server_negotiation.cc:407] Connection header received
0928 02:18:11.163491 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:11.163498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:11.163611 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:11.163716 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:11.165404 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.166562 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.168669 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.169359 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.169490 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.174152 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:11.174159 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:11.174165 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:11.174210 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:11.178035 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.179074 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.179080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.179083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.179240 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.179700 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.179704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.179708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.180090 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:11.180252 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:11.180850 (+ 598us) server_negotiation.cc:299] Negotiation successful
0928 02:18:11.181091 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:18:11.181933 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.162107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:45809 (local address 127.0.0.1:53278)
0928 02:18:11.162917 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:11.162953 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:11.163245 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:11.163781 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:11.163798 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:11.164772 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:11.165219 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.165261 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.166697 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.166705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.168491 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.168509 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.169541 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.169550 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.169796 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.170560 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:11.170587 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:11.173884 (+ 3297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:11.178222 (+ 4338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.178238 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.178249 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.178926 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.179366 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.179374 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.179380 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.179565 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.180237 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:11.180250 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:11.180485 (+ 235us) client_negotiation.cc:769] Sending connection context
0928 02:18:11.180718 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 02:18:11.181149 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":352,"threads_started":1}
I20260928 02:18:11.183532 11333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:18:11.183815 11333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } }
I20260928 02:18:11.184394 10798 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.140.253:45809: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1878aa81226947478f7f8470d8f0452f"
member_type: NON_VOTER
last_known_addr {
host: "127.4.140.253"
port: 45809
}
attrs {
promote: true
}
}
dest_uuid: "4858ab829e994bd4a9c26227f4e81197"
cas_config_opid_index: -1
I20260928 02:18:11.185274 11333 leader_election.cc:288] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4858ab829e994bd4a9c26227f4e81197 (127.4.140.254:36419)
I20260928 02:18:11.185439 10798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: NON_VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: true } }
I20260928 02:18:11.186012 10808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1878aa81226947478f7f8470d8f0452f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4858ab829e994bd4a9c26227f4e81197" is_pre_election: true
I20260928 02:18:11.187232 11278 leader_election.cc:302] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1878aa81226947478f7f8470d8f0452f; no voters: 4858ab829e994bd4a9c26227f4e81197
I20260928 02:18:11.187996 11333 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.140.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.252.keytab.
Entry for principal oryx/127.4.140.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.252.keytab.
I20260928 02:18:11.194144 11301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:11.195855 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.252:45985
--webserver_interface=127.4.140.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.252.keytab
--principal=oryx/127.4.140.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:36419,127.4.140.253:45809,127.4.140.252:45985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:18:11.198221 11301 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 1 FOLLOWER]: Refusing update from remote peer 4858ab829e994bd4a9c26227f4e81197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 02:18:11.193181 11356 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1878aa81226947478f7f8470d8f0452f (127.4.140.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: NON_VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: true } } }
I20260928 02:18:11.203151 11357 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: NON_VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:18:11.207993 11356 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.140.253:45809
I20260928 02:18:11.208675 11259 master.cc:537] Master@127.4.140.253:45809 shutting down...
I20260928 02:18:11.208966 11359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: NON_VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: true } } }
I20260928 02:18:11.209302 11359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:11.238031 11301 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1878aa81226947478f7f8470d8f0452f (127.4.140.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: NON_VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: true } } }
I20260928 02:18:11.242865 11366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: NON_VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: true } } }
I20260928 02:18:11.243342 11366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: This master's current role is: LEARNER
I20260928 02:18:11.243686 11333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4858ab829e994bd4a9c26227f4e81197. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: NON_VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: true } } }
I20260928 02:18:11.243937 11333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: This master's current role is: LEARNER
I20260928 02:18:11.244184 11363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: NON_VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: true } } }
I20260928 02:18:11.244340 11363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: This master's current role is: LEARNER
I20260928 02:18:11.266000 11259 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 1 LEARNER]: Raft consensus shutting down.
I20260928 02:18:11.266537 11259 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 1 LEARNER]: Raft consensus is shut down!
I20260928 02:18:11.266732 11259 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f: stopping tablet replica
I20260928 02:18:11.273977 11259 master.cc:552] Master@127.4.140.253:45809 shutdown complete.
I20260928 02:18:11.274075 11259 master_runner.cc:305] Clearing existing system tablet
I20260928 02:18:11.274490 11259 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:11.274590 11259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:11.277593 11259 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 02:18:11.279250 11368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:11.279590 11259 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:11.280025 11259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:18:11.280160 11259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/wal
uuid: "1878aa81226947478f7f8470d8f0452f"
format_stamp: "Formatted at 2026-09-28 02:18:10 on dist-test-slave-h37r"
server_key: "8c2fe841670435c05964eafa354ce724"
server_key_iv: "300cb08b1e44aa76eda25db1e995b218"
server_key_version: "encryptionkey@0"
I20260928 02:18:11.284786 11259 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:11.291911 11259 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:18:11.292217 11259 log.cc:1196] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 02:18:11.292757 11259 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f: Deleting consensus metadata
I20260928 02:18:11.293078 11259 master_runner.cc:315] Copying system tablet from 127.4.140.254:36419
I20260928 02:18:11.296000 11259 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f: tablet copy: Beginning tablet copy session from remote peer at address 127.4.140.254:36419
I20260928 02:18:11.318921 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.296883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.0.0.1:46778)
0928 02:18:11.304397 (+ 7514us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:11.304427 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:11.304629 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:11.305090 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:11.305095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:11.305614 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:11.305891 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.305902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.307357 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.307363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.308978 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.308993 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.309935 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.309941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.310127 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.310849 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:11.310872 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:11.312045 (+ 1173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:11.315951 (+ 3906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.315957 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.315961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.316451 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.316917 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.316923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.316927 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.317076 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.317761 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:11.317766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:11.317970 (+ 204us) client_negotiation.cc:769] Sending connection context
0928 02:18:11.318128 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 02:18:11.318432 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7346,"thread_start_us":7204,"threads_started":1}
I20260928 02:18:11.318939 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.296914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46778 (local address 127.4.140.254:36419)
0928 02:18:11.297104 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:11.297110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:11.304584 (+ 7474us) server_negotiation.cc:407] Connection header received
0928 02:18:11.304787 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:11.304793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:11.304894 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:11.305025 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:11.306069 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.307182 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.309141 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.309774 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.309906 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.312261 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:11.312268 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:11.312273 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:11.312312 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:11.315788 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.316594 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.316603 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.316607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.316802 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.317228 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.317233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.317236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.317615 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:11.317768 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:11.318294 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 02:18:11.318505 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:18:11.320320 10818 tablet_copy_service.cc:140] P 4858ab829e994bd4a9c26227f4e81197: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1878aa81226947478f7f8470d8f0452f ({username='oryx', principal='oryx/127.4.140.253@KRBTEST.COM'} at 127.0.0.1:46778)
I20260928 02:18:11.320585 10818 tablet_copy_service.cc:161] P 4858ab829e994bd4a9c26227f4e81197: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1878aa81226947478f7f8470d8f0452f at {username='oryx', principal='oryx/127.4.140.253@KRBTEST.COM'} at 127.0.0.1:46778: session id = 1878aa81226947478f7f8470d8f0452f-00000000000000000000000000000000
I20260928 02:18:11.325412 10818 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:18:11.327600 11259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:18:11.334263 11259 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f: tablet copy: Starting download of 0 data blocks...
I20260928 02:18:11.334584 11259 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f: tablet copy: Starting download of 1 WAL segments...
I20260928 02:18:11.337251 11259 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:18:11.341603 10818 tablet_copy_service.cc:342] P 4858ab829e994bd4a9c26227f4e81197: Request end of tablet copy session 1878aa81226947478f7f8470d8f0452f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.140.253@KRBTEST.COM'} at 127.0.0.1:46778
I20260928 02:18:11.341787 10818 tablet_copy_service.cc:433] P 4858ab829e994bd4a9c26227f4e81197: ending tablet copy session 1878aa81226947478f7f8470d8f0452f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1878aa81226947478f7f8470d8f0452f
I20260928 02:18:11.352454 11259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:11.355535 11375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:11.357155 11376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:11.364316 11259 server_base.cc:1034] running on GCE node
W20260928 02:18:11.364648 11379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:11.365131 11259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:11.365881 11259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:11.367154 11259 hybrid_clock.cc:648] HybridClock initialized: now 1790561891367077 us; error 56 us; skew 500 ppm
I20260928 02:18:11.367401 11259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:11 dist-test-slave-h37r krb5kdc[10753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561891, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:11.369656 11259 init.cc:377] Logged in from keytab as oryx/127.4.140.253@KRBTEST.COM (short username oryx)
I20260928 02:18:11.370975 11259 webserver.cc:460] Webserver started at http://127.4.140.253:33293/ using document root <none> and password file <none>
I20260928 02:18:11.371310 11259 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:11.371387 11259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:11.374702 11259 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:18:11.376747 11385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:11.377202 11259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:18:11.377344 11259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/wal
uuid: "1878aa81226947478f7f8470d8f0452f"
format_stamp: "Formatted at 2026-09-28 02:18:10 on dist-test-slave-h37r"
server_key: "8c2fe841670435c05964eafa354ce724"
server_key_iv: "300cb08b1e44aa76eda25db1e995b218"
server_key_version: "encryptionkey@0"
I20260928 02:18:11.377512 11259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:11.392338 11259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:11.396538 11259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:11.431869 11259 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.253:45809
I20260928 02:18:11.431955 11436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.253:45809 every 8 connection(s)
I20260928 02:18:11.432940 11259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-1/data/info.pb
I20260928 02:18:11.438354 11437 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:18:11.442396 11437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f: Bootstrap starting.
I20260928 02:18:11.456385 11437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:18:11.456923 11437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f: Bootstrap complete.
I20260928 02:18:11.458007 11437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: NON_VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: true } }
I20260928 02:18:11.458252 11437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1878aa81226947478f7f8470d8f0452f, State: Initialized, Role: LEARNER
I20260928 02:18:11.458525 11437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: NON_VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: true } }
I20260928 02:18:11.459343 11440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: NON_VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: true } } }
I20260928 02:18:11.459609 11437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:18:11.459748 11440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: This master's current role is: LEARNER
I20260928 02:18:11.467177 11451 catalog_manager.cc:1259] Loaded cluster ID: dfd33f39829443b9ac381d5072a1c40f
I20260928 02:18:11.467406 11451 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f: loading cluster ID for follower catalog manager: success
I20260928 02:18:11.471141 11451 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f: acquiring CA information for follower catalog manager: success
I20260928 02:18:11.474068 11451 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:18:11.557307 11254 heartbeater.cc:503] Master 127.4.140.254:36419 was elected leader, sending a full tablet report...
W20260928 02:18:11.600268 11358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:11.600557 11358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:11.600723 11358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:11.608263 11358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:18:11.608385 11358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:11.608464 11358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:11.608556 11358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:18:11.608630 11358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:18:11.618562 11358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:36419,127.4.140.253:45809,127.4.140.252:45985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.252.keytab
--principal=oryx/127.4.140.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.252:45985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.140.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:11.619908 11358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:11.621212 11358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:11.632591 11456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:11.632591 11457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:11.632817 11459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:11.633647 11358 server_base.cc:1034] running on GCE node
I20260928 02:18:11.634434 11358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:11.635828 11358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:11.637074 11358 hybrid_clock.cc:648] HybridClock initialized: now 1790561891637038 us; error 28 us; skew 500 ppm
I20260928 02:18:11.637398 11358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:11 dist-test-slave-h37r krb5kdc[10753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561891, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:11.640298 11358 init.cc:377] Logged in from keytab as oryx/127.4.140.252@KRBTEST.COM (short username oryx)
I20260928 02:18:11.641822 11358 webserver.cc:460] Webserver started at http://127.4.140.252:37185/ using document root <none> and password file <none>
I20260928 02:18:11.642485 11358 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:11.642621 11358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:11.642915 11358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:11.645793 11358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/data/instance:
uuid: "e8106b15bf894b04b0413a7a07fc5198"
format_stamp: "Formatted at 2026-09-28 02:18:11 on dist-test-slave-h37r"
server_key: "46de916442c613584a811058c4258ee2"
server_key_iv: "ffd1bd372fcd1c53499815d320da4e6b"
server_key_version: "encryptionkey@0"
I20260928 02:18:11.646481 11358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/wal/instance:
uuid: "e8106b15bf894b04b0413a7a07fc5198"
format_stamp: "Formatted at 2026-09-28 02:18:11 on dist-test-slave-h37r"
server_key: "46de916442c613584a811058c4258ee2"
server_key_iv: "ffd1bd372fcd1c53499815d320da4e6b"
server_key_version: "encryptionkey@0"
I20260928 02:18:11.651274 11358 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 02:18:11.654560 11466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:11.656100 11358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:18:11.656255 11358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/wal
uuid: "e8106b15bf894b04b0413a7a07fc5198"
format_stamp: "Formatted at 2026-09-28 02:18:11 on dist-test-slave-h37r"
server_key: "46de916442c613584a811058c4258ee2"
server_key_iv: "ffd1bd372fcd1c53499815d320da4e6b"
server_key_version: "encryptionkey@0"
I20260928 02:18:11.656446 11358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:11.697103 11358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:11.701408 11358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:11.701727 11358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:11.720113 11358 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.252:45985
I20260928 02:18:11.720144 11517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.252:45985 every 8 connection(s)
I20260928 02:18:11.721457 11358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/data/info.pb
I20260928 02:18:11.725149 11518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:11.726536 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 11358
I20260928 02:18:11.734694 11518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:18:11 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.140.252@KRBTEST.COM
Sep 28 02:18:11 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561891, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.252@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM
I20260928 02:18:11.751164 11521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.728715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52162 (local address 127.4.140.252:45985)
0928 02:18:11.729504 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:11.729516 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:11.729555 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:18:11.731540 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:11.731562 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:11.732046 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:11.732384 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:11.734226 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.735644 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.737287 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.738402 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.738587 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.743375 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:11.743384 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:11.743399 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:11.743485 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:11.746774 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.747454 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.747464 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.747477 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.747634 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.748071 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.748077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.748081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.748777 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:11.749090 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:11.749408 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 02:18:11.749833 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"spinlock_wait_cycles":3968,"thread_start_us":376,"threads_started":1}
I20260928 02:18:11.752727 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.737508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46784 (local address 127.4.140.254:36419)
0928 02:18:11.737728 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:11.737737 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:11.738416 (+ 679us) server_negotiation.cc:407] Connection header received
0928 02:18:11.738596 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:11.738601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:11.738683 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:11.738790 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:11.739791 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.740981 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.742294 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.742954 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.743095 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.747025 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:11.747030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:11.747034 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:11.747065 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:11.750065 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.750727 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.750732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.750735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.750885 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.751228 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.751232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.751236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.751587 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:11.751731 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:11.752236 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 02:18:11.752457 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:18:11.752728 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.737453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.0.0.1:46784)
0928 02:18:11.738258 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:11.738314 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:11.738475 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:11.738842 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:11.738847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:11.739409 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:11.739640 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.739653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.741126 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.741129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.742143 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.742154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.743160 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.743165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.743365 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.744120 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:11.744143 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:11.746852 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:11.750183 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.750192 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.750197 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.750618 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.750991 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.750995 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.750997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.751122 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.751710 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:11.751715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:11.751860 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 02:18:11.752115 (+ 255us) client_negotiation.cc:240] Negotiation successful
0928 02:18:11.752343 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"spinlock_wait_cycles":1408,"thread_start_us":400,"threads_started":1}
I20260928 02:18:11.754118 11518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:18:11 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561891, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.252@KRBTEST.COM for oryx/127.4.140.253@KRBTEST.COM
I20260928 02:18:11.767118 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.754873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:45809 (local address 127.0.0.1:53292)
0928 02:18:11.755059 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:11.755075 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:11.755160 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:11.755657 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:11.755660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:11.756007 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:11.756133 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.756140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.757353 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.757356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.758261 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.758270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.758928 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.758932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.759048 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.759615 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:11.759632 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:11.761679 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:11.764812 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.764816 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.764819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.765252 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.765591 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.765594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.765596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.765718 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.766313 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:11.766316 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:11.766476 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 02:18:11.766590 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:18:11.766800 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:18:11.767242 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.754904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.4.140.253:45809)
0928 02:18:11.755419 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:11.755423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:11.755437 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:18:11.755484 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:11.755488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:11.755543 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:11.755626 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:11.756251 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.757234 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.758385 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.758800 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.758905 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.761885 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:11.761890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:11.761893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:11.761924 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:11.764692 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.765374 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.765377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.765380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.765494 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.765801 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.765804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.765806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.766113 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:11.766261 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:11.766693 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 02:18:11.766848 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":281,"threads_started":1}
I20260928 02:18:11.768364 11518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:18:11 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561891, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.252@KRBTEST.COM for oryx/127.4.140.252@KRBTEST.COM
I20260928 02:18:11.780385 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.769272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:45985 (local address 127.0.0.1:52164)
0928 02:18:11.769438 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:11.769456 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:11.769568 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:11.769825 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:11.769828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:11.770146 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:11.770277 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.770284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.771180 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.771183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.772138 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.772149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.772895 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.772899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.773060 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.773548 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:11.773563 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:11.775496 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:11.778219 (+ 2723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.778223 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.778232 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.778686 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.779029 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.779033 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.779035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.779176 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.779602 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:11.779605 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:11.779719 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 02:18:11.779837 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 02:18:11.780085 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 02:18:11.780453 11521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.769069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52164 (local address 127.4.140.252:45985)
0928 02:18:11.769215 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:11.769219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:11.769526 (+ 307us) server_negotiation.cc:407] Connection header received
0928 02:18:11.769651 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:11.769655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:11.769712 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:11.769785 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:11.770388 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.771037 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.772275 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.772768 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.772858 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.775673 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:11.775678 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:11.775681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:11.775707 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:11.777972 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.778801 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.778804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.778807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.778916 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.779262 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.779265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.779267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.779508 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:11.779617 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:11.779968 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 02:18:11.780208 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":50}
I20260928 02:18:11.785094 11518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: Bootstrap starting.
I20260928 02:18:11.788241 11518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:11.789250 11518 log.cc:824] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:11.791874 11518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: No bootstrap required, opened a new log
I20260928 02:18:11.798866 11518 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } }
I20260928 02:18:11.799225 11518 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:11.799350 11518 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8106b15bf894b04b0413a7a07fc5198, State: Initialized, Role: FOLLOWER
I20260928 02:18:11.799947 11518 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } }
I20260928 02:18:11.800799 11525 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } } }
I20260928 02:18:11.801111 11525 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:18:11.801781 11518 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:18:11.806439 11531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: NON_VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: true } } }
I20260928 02:18:11.806892 11531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:11.809983 11539 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:18:11.817269 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.807138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53294 (local address 127.4.140.253:45809)
0928 02:18:11.807473 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:11.807483 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:11.811380 (+ 3897us) server_negotiation.cc:407] Connection header received
0928 02:18:11.811550 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:11.811553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:11.811647 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:11.811775 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:11.812311 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.813284 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.815224 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.816339 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.816446 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.816629 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:11.816869 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 02:18:11.816989 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260928 02:18:11.817297 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.806639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:45809 (local address 127.0.0.1:53294)
0928 02:18:11.811241 (+ 4602us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:11.811263 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:11.811425 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:11.811811 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:11.811816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:11.811846 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:11.812127 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.812142 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.813427 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.813432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.815064 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.815078 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.816501 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.816507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.816686 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.816750 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 02:18:11.816854 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:18:11.816988 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4459,"thread_start_us":4281,"threads_started":1}
I20260928 02:18:11.818101 11440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4858ab829e994bd4a9c26227f4e81197. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: NON_VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: true } } }
I20260928 02:18:11.818344 11440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: This master's current role is: LEARNER
I20260928 02:18:11.818886 11531 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197: attempting to promote NON_VOTER 1878aa81226947478f7f8470d8f0452f to VOTER
I20260928 02:18:11.819141 11358 master_runner.cc:428] Detected that this master e8106b15bf894b04b0413a7a07fc5198 is joining an existing cluster
I20260928 02:18:11.819264 11358 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.140.252:45985
I20260928 02:18:11.819705 11531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } }
I20260928 02:18:11.821401 11412 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 1 LEARNER]: Refusing update from remote peer 4858ab829e994bd4a9c26227f4e81197: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 02:18:11.821921 11532 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 02:18:11.825668 11538 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1878aa81226947478f7f8470d8f0452f (127.4.140.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } }
I20260928 02:18:11.831593 11412 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1878aa81226947478f7f8470d8f0452f (127.4.140.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } }
I20260928 02:18:11.836612 11440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } }
I20260928 02:18:11.836834 11440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:18:11.837441 11538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } }
I20260928 02:18:11.837522 11531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } }
I20260928 02:18:11.837882 11538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:11.837893 11531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:11.840204 11525 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:18:11.840849 11525 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } }
I20260928 02:18:11.842746 11525 leader_election.cc:288] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4858ab829e994bd4a9c26227f4e81197 (127.4.140.254:36419), 1878aa81226947478f7f8470d8f0452f (127.4.140.253:45809)
I20260928 02:18:11.843508 11412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8106b15bf894b04b0413a7a07fc5198" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1878aa81226947478f7f8470d8f0452f" is_pre_election: true
I20260928 02:18:11.844525 10808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8106b15bf894b04b0413a7a07fc5198" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4858ab829e994bd4a9c26227f4e81197" is_pre_election: true
I20260928 02:18:11.845611 11470 leader_election.cc:302] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8106b15bf894b04b0413a7a07fc5198; no voters: 1878aa81226947478f7f8470d8f0452f, 4858ab829e994bd4a9c26227f4e81197
I20260928 02:18:11.846299 11525 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:18:11.855335 11521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.823632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52168 (local address 127.4.140.252:45985)
0928 02:18:11.824083 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:11.824091 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:11.840045 (+ 15954us) server_negotiation.cc:407] Connection header received
0928 02:18:11.840492 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:11.840497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:11.840556 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:11.840662 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:11.842370 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.843478 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.845874 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.846583 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.846667 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.849089 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:11.849095 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:11.849099 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:11.849135 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:11.852557 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.853297 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.853302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.853305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.853451 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.853850 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.853862 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.853866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.854160 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:11.854268 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:11.854635 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 02:18:11.854959 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":59}
I20260928 02:18:11.855373 11548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.824429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:45985 (local address 127.0.0.1:52168)
0928 02:18:11.839885 (+ 15456us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:11.839924 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:11.840056 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:11.840675 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:11.840679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:11.841697 (+ 1018us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:11.842198 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.842213 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.843843 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.843848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.845686 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.845698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.846949 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.846953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.847057 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.847698 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:11.847732 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:11.848904 (+ 1172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:11.852704 (+ 3800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.852709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.852714 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.853176 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.853568 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.853573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.853576 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.853734 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.854314 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:11.854317 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:11.854419 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 02:18:11.854546 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 02:18:11.854832 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15337,"mutex_wait_us":65,"spinlock_wait_cycles":13696,"thread_start_us":15149,"threads_started":1}
I20260928 02:18:11.855827 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.823846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.4.140.253:45809)
0928 02:18:11.824020 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:11.824025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:11.839209 (+ 15184us) server_negotiation.cc:407] Connection header received
0928 02:18:11.839437 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:11.839446 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:11.839550 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:11.839679 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:11.842547 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.843683 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.846207 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.846864 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.846954 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.850188 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:11.850193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:11.850202 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:11.850234 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:11.852975 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.853655 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.853665 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.853668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.853777 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.854164 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.854169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.854172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.854514 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:11.854671 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:11.855247 (+ 576us) server_negotiation.cc:299] Negotiation successful
0928 02:18:11.855490 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:18:11.856896 11547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.824262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:45809 (local address 127.0.0.1:53300)
0928 02:18:11.839053 (+ 14791us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:11.839082 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:11.839283 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:11.839945 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:11.839952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:11.841545 (+ 1593us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:11.842358 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.842371 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.844226 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.844234 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.846045 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.846059 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.847269 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.847275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.847539 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.848415 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:11.848447 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:11.849964 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:11.853101 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.853108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.853112 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.853563 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.853888 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.853893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.853896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.854054 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.854723 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:11.854727 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:11.854955 (+ 228us) client_negotiation.cc:769] Sending connection context
0928 02:18:11.856090 (+ 1135us) client_negotiation.cc:240] Negotiation successful
0928 02:18:11.856415 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14541,"mutex_wait_us":158,"spinlock_wait_cycles":100864,"thread_start_us":14363,"threads_started":1}
I20260928 02:18:11.857148 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.823415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.4.140.254:36419)
0928 02:18:11.823607 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:11.823613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:11.841231 (+ 17618us) server_negotiation.cc:407] Connection header received
0928 02:18:11.841468 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:11.841473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:11.841561 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:11.841634 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:11.843047 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.844178 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.846208 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.846864 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.846954 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.850188 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:11.850198 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:11.850203 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:11.850266 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:11.853765 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.854707 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.854711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.854715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.854866 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.855361 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.855366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.855368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.855734 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:11.855867 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:11.856540 (+ 673us) server_negotiation.cc:299] Negotiation successful
0928 02:18:11.856753 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:18:11.858070 11549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.824572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.0.0.1:46786)
0928 02:18:11.841121 (+ 16549us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:11.841143 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:11.841324 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:11.841823 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:11.841833 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:11.842530 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:11.842878 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.842889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.844347 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.844351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.846054 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.846061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.847263 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.847269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.847544 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.848488 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:11.848507 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:11.849964 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:11.854177 (+ 4213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.854182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.854185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.854583 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.855011 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.855028 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.855032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.855229 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.855985 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:11.855990 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:11.856235 (+ 245us) client_negotiation.cc:769] Sending connection context
0928 02:18:11.857308 (+ 1073us) client_negotiation.cc:240] Negotiation successful
0928 02:18:11.857611 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16410,"mutex_wait_us":157,"spinlock_wait_cycles":18432,"thread_start_us":15820,"threads_started":1}
I20260928 02:18:11.858407 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 10859
Sep 28 02:18:11 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561888, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.254@KRBTEST.COM for oryx/127.4.140.252@KRBTEST.COM
Sep 28 02:18:11 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.140.193@KRBTEST.COM
I20260928 02:18:11.876035 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.863050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:45985 (local address 127.0.0.1:52184)
0928 02:18:11.863215 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:11.863232 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:11.863321 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:11.863677 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:11.863680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:11.864084 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:11.864262 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.864271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.865246 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.865252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.866492 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.866507 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.867338 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.867344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.867525 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.868069 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:11.868085 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:11.870326 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:11.873612 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.873619 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.873625 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.874099 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.874470 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.874473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.874475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.874619 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.875227 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:11.875230 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:11.875368 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 02:18:11.875511 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 02:18:11.875703 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 02:18:11.876962 11521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.863125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52184 (local address 127.4.140.252:45985)
0928 02:18:11.863279 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:11.863284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:11.863302 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:18:11.863435 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:11.863439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:11.863566 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:11.863666 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:11.864399 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.865122 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.866620 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.867191 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.867339 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.870505 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:11.870512 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:11.870517 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:11.870560 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:11.873489 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.874205 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.874212 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.874216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.874368 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.874743 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.874747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.874752 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.875109 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:11.875272 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:11.876084 (+ 812us) server_negotiation.cc:299] Negotiation successful
0928 02:18:11.876617 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:18:11.878334 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.859447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53278 (local address 127.4.140.193:35389)
0928 02:18:11.860231 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:11.860237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:11.860260 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:18:11.860316 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:11.860322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:11.860934 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:11.861112 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:11.862817 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.864467 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.865862 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.866740 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.866833 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.871851 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:11.871862 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:11.871877 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:11.871953 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:11.875407 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.876131 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.876143 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.876146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.876281 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.876692 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.876697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.876700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.877121 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:11.877444 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:11.877755 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 02:18:11.878014 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":385,"threads_started":1}
I20260928 02:18:11.878569 10797 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.140.252:45985: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198"
member_type: NON_VOTER
last_known_addr {
host: "127.4.140.252"
port: 45985
}
attrs {
promote: true
}
}
dest_uuid: "4858ab829e994bd4a9c26227f4e81197"
cas_config_opid_index: 6
I20260928 02:18:11.879426 10797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: NON_VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: true } }
I20260928 02:18:11.879794 10982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:11.881639 11412 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 1 FOLLOWER]: Refusing update from remote peer 4858ab829e994bd4a9c26227f4e81197: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 02:18:11.882074 11493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:11.882190 11538 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 02:18:11.884282 11531 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e8106b15bf894b04b0413a7a07fc5198 (127.4.140.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: NON_VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: true } } }
I20260928 02:18:11.886734 11531 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.140.252:45985
I20260928 02:18:11.886855 11493 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 1 FOLLOWER]: Refusing update from remote peer 4858ab829e994bd4a9c26227f4e81197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 02:18:11.887041 11531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: NON_VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: true } } }
I20260928 02:18:11.887261 11531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:11.887562 11538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: NON_VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: true } } }
I20260928 02:18:11.887807 11538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:11.887915 11358 master.cc:537] Master@127.4.140.252:45985 shutting down...
I20260928 02:18:11.888857 11538 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: NON_VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:18:11.888609 11412 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e8106b15bf894b04b0413a7a07fc5198 (127.4.140.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: NON_VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: true } } }
W20260928 02:18:11.891180 10783 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260928 02:18:11.892577 11440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: NON_VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: true } } }
I20260928 02:18:11.892822 11440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:18:11.893342 10783 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 -> Peer e8106b15bf894b04b0413a7a07fc5198 (127.4.140.252:45985): Couldn't send request to peer e8106b15bf894b04b0413a7a07fc5198. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:18:11.898099 11358 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:18:11.898470 11358 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:11.898670 11358 tablet_replica.cc:330] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: stopping tablet replica
I20260928 02:18:11.914551 11358 master.cc:552] Master@127.4.140.252:45985 shutdown complete.
I20260928 02:18:11.914635 11358 master_runner.cc:305] Clearing existing system tablet
I20260928 02:18:11.914969 11358 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:11.915081 11358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:11.917263 11358 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:18:11.918581 11555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:11.918934 11358 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:11.919180 11358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:18:11.919269 11358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/wal
uuid: "e8106b15bf894b04b0413a7a07fc5198"
format_stamp: "Formatted at 2026-09-28 02:18:11 on dist-test-slave-h37r"
server_key: "46de916442c613584a811058c4258ee2"
server_key_iv: "ffd1bd372fcd1c53499815d320da4e6b"
server_key_version: "encryptionkey@0"
I20260928 02:18:11.924203 11358 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:11.930867 11358 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:18:11.931046 11358 log.cc:1196] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 02:18:11.931525 11358 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: Deleting consensus metadata
I20260928 02:18:11.931777 11358 master_runner.cc:315] Copying system tablet from 127.4.140.254:36419
I20260928 02:18:11.937451 11358 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: tablet copy: Beginning tablet copy session from remote peer at address 127.4.140.254:36419
I20260928 02:18:11.950614 11561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.938188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.0.0.1:46792)
0928 02:18:11.938634 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:11.938650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:11.938770 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:11.939149 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:11.939152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:11.939492 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:11.939659 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.939667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.941172 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.941177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.942028 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:11.942038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.943102 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.943106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:11.943221 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.943865 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:11.943882 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:11.944607 (+ 725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:11.948431 (+ 3824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.948435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.948437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.948764 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.949161 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:11.949164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:11.949167 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:11.949286 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:11.949872 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:11.949877 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:11.950013 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 02:18:11.950127 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:18:11.950327 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":242,"threads_started":1}
I20260928 02:18:11.950861 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:11.938310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46792 (local address 127.4.140.254:36419)
0928 02:18:11.938555 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:11.938563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:11.938736 (+ 173us) server_negotiation.cc:407] Connection header received
0928 02:18:11.938889 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:11.938896 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:11.938981 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:11.939102 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:11.939820 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.941032 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.942206 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:11.942931 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:11.943088 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:11.944797 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:11.944805 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:11.944811 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:11.944852 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:11.948309 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.948882 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.948889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.948894 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.949058 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:11.949385 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:11.949390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:11.949394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:11.949740 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:11.949904 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:11.950247 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 02:18:11.950479 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260928 02:18:11.951361 10818 tablet_copy_service.cc:140] P 4858ab829e994bd4a9c26227f4e81197: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e8106b15bf894b04b0413a7a07fc5198 ({username='oryx', principal='oryx/127.4.140.252@KRBTEST.COM'} at 127.0.0.1:46792)
I20260928 02:18:11.951520 10818 tablet_copy_service.cc:161] P 4858ab829e994bd4a9c26227f4e81197: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e8106b15bf894b04b0413a7a07fc5198 at {username='oryx', principal='oryx/127.4.140.252@KRBTEST.COM'} at 127.0.0.1:46792: session id = e8106b15bf894b04b0413a7a07fc5198-00000000000000000000000000000000
I20260928 02:18:11.953788 10818 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:18:11.955541 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:18:11.962378 11358 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: tablet copy: Starting download of 0 data blocks...
I20260928 02:18:11.962694 11358 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: tablet copy: Starting download of 1 WAL segments...
I20260928 02:18:11.965191 11358 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:18:11.969633 10818 tablet_copy_service.cc:342] P 4858ab829e994bd4a9c26227f4e81197: Request end of tablet copy session e8106b15bf894b04b0413a7a07fc5198-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.140.252@KRBTEST.COM'} at 127.0.0.1:46792
I20260928 02:18:11.969750 10818 tablet_copy_service.cc:433] P 4858ab829e994bd4a9c26227f4e81197: ending tablet copy session e8106b15bf894b04b0413a7a07fc5198-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e8106b15bf894b04b0413a7a07fc5198
I20260928 02:18:11.980830 11358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:11.983522 11563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:11.983887 11562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:11.984462 11565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:11.990931 11358 server_base.cc:1034] running on GCE node
I20260928 02:18:11.991295 11358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:11.991994 11358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:11.993228 11358 hybrid_clock.cc:648] HybridClock initialized: now 1790561891993158 us; error 33 us; skew 500 ppm
I20260928 02:18:11.993479 11358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:11 dist-test-slave-h37r krb5kdc[10753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561891, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:11.995357 11358 init.cc:377] Logged in from keytab as oryx/127.4.140.252@KRBTEST.COM (short username oryx)
I20260928 02:18:11.996623 11358 webserver.cc:460] Webserver started at http://127.4.140.252:33015/ using document root <none> and password file <none>
I20260928 02:18:11.996903 11358 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:11.997011 11358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:11.999109 11358 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:18:12.000619 11572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:12.000990 11358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:18:12.001132 11358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/wal
uuid: "e8106b15bf894b04b0413a7a07fc5198"
format_stamp: "Formatted at 2026-09-28 02:18:11 on dist-test-slave-h37r"
server_key: "46de916442c613584a811058c4258ee2"
server_key_iv: "ffd1bd372fcd1c53499815d320da4e6b"
server_key_version: "encryptionkey@0"
I20260928 02:18:12.001295 11358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:12.006588 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 10859
I20260928 02:18:12.021420 11358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:12.025823 11358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:12.029353 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:35389
--local_ip_for_outbound_sockets=127.4.140.193
--tserver_master_addrs=127.4.140.254:36419
--webserver_port=41515
--webserver_interface=127.4.140.193
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.193.keytab
--principal=oryx/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.140.254:36419,127.4.140.253:45809
--tserver_master_addrs=127.4.140.254:36419,127.4.140.253:45809,127.4.140.252:45985 with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:18:12.047449 11358 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.252:45985
I20260928 02:18:12.047464 11624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.252:45985 every 8 connection(s)
I20260928 02:18:12.048632 11358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/master-2/data/info.pb
I20260928 02:18:12.052074 11625 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:18:12.054261 11625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: Bootstrap starting.
I20260928 02:18:12.068943 11625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:18:12.069365 11625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: Bootstrap complete.
I20260928 02:18:12.070154 11625 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: NON_VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: true } }
I20260928 02:18:12.070371 11625 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e8106b15bf894b04b0413a7a07fc5198, State: Initialized, Role: LEARNER
I20260928 02:18:12.070592 11625 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: NON_VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: true } }
I20260928 02:18:12.071312 11628 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: NON_VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: true } } }
I20260928 02:18:12.071643 11628 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:18:12.071661 11625 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:18:12.082798 11639 catalog_manager.cc:1259] Loaded cluster ID: dfd33f39829443b9ac381d5072a1c40f
I20260928 02:18:12.082916 11639 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: loading cluster ID for follower catalog manager: success
I20260928 02:18:12.086472 11639 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: acquiring CA information for follower catalog manager: success
I20260928 02:18:12.088810 11639 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:18:12.318892 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:12.310144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:45985 (local address 127.0.0.1:52198)
0928 02:18:12.310425 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:12.310448 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:12.310607 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:12.311706 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:12.311713 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:12.311746 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:12.311978 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:12.311991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:12.314213 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:12.314219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:12.316198 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:12.316217 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:12.318020 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:12.318026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:12.318243 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:12.318277 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:18:12.318382 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:18:12.318548 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260928 02:18:12.318975 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:12.310335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52198 (local address 127.4.140.252:45985)
0928 02:18:12.311250 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:12.311259 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:12.311287 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:18:12.311372 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:12.311379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:12.311480 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:12.311638 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:12.312172 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:12.314044 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:12.316383 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:12.317824 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:12.317984 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:12.318271 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:12.318426 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 02:18:12.318591 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":570,"threads_started":1}
I20260928 02:18:12.320660 11600 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260928 02:18:12.321285 11628 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4858ab829e994bd4a9c26227f4e81197. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: NON_VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: true } } }
I20260928 02:18:12.321609 11628 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [sys.catalog]: This master's current role is: LEARNER
W20260928 02:18:12.377663 11590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:12.378054 11590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:12.378212 11590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:12.386111 11590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:12.386312 11590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:12.386548 11590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:18:12.397967 11590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.193.keytab
--principal=oryx/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:35389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=41515
--tserver_master_addrs=127.4.140.254:36419,127.4.140.253:45809,127.4.140.252:45985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:12.399425 11590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:12.400935 11590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:12.415987 11646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:12.416375 11647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:12.416625 11649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:12.417725 11590 server_base.cc:1034] running on GCE node
I20260928 02:18:12.418300 11590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:12.419237 11590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:12.420464 11590 hybrid_clock.cc:648] HybridClock initialized: now 1790561892420399 us; error 56 us; skew 500 ppm
I20260928 02:18:12.420795 11590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:12 dist-test-slave-h37r krb5kdc[10753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561892, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:12.423877 11590 init.cc:377] Logged in from keytab as oryx/127.4.140.193@KRBTEST.COM (short username oryx)
I20260928 02:18:12.425393 11590 webserver.cc:460] Webserver started at http://127.4.140.193:41515/ using document root <none> and password file <none>
I20260928 02:18:12.426082 11590 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:12.426224 11590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:12.431746 11590 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:18:12.434528 11656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:12.436179 11590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:18:12.436342 11590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "1d4a4eb5099e4262a21d9704f43f0bf1"
format_stamp: "Formatted at 2026-09-28 02:18:09 on dist-test-slave-h37r"
server_key: "b012f95b49c9b0ab8548b4b06d1d61db"
server_key_iv: "fd71463e303cbe10050dc0e6505561b1"
server_key_version: "encryptionkey@0"
I20260928 02:18:12.437652 11590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:12.492101 11590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:12.496734 11590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:12.497030 11590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:12.498090 11590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:12.499440 11590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:12.499571 11590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:12.499689 11590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:12.499773 11590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:12.534488 11590 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:35389
I20260928 02:18:12.534578 11768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:35389 every 8 connection(s)
I20260928 02:18:12.536057 11590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:18:12.538833 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 11590
I20260928 02:18:12.539152 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 10993
Sep 28 02:18:12 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561892, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.253@KRBTEST.COM
Sep 28 02:18:12 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561892, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.252@KRBTEST.COM
Sep 28 02:18:12 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561892, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM
Sep 28 02:18:12 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.140.194@KRBTEST.COM
I20260928 02:18:12.583040 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:12.538983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:56093 (local address 127.4.140.252:45985)
0928 02:18:12.539320 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:12.539327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:12.559423 (+ 20096us) server_negotiation.cc:407] Connection header received
0928 02:18:12.559610 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:12.559616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:12.559699 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:12.559799 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:12.563154 (+ 3355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:12.564346 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:12.567753 (+ 3407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:12.568427 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:12.568540 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:12.574779 (+ 6239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:12.574787 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:12.574793 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:12.574836 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:12.579038 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:12.580134 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:12.580140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:12.580145 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:12.580287 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:12.580819 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:12.580824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:12.580830 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:12.581210 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:12.581328 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:12.582448 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0928 02:18:12.582686 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260928 02:18:12.583154 11773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:12.539012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:50727 (local address 127.4.140.253:45809)
0928 02:18:12.560772 (+ 21760us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:12.560779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:12.560804 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:18:12.560896 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:12.560902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:12.560988 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:12.561106 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:12.563270 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:12.564467 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:12.567991 (+ 3524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:12.568628 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:12.568734 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:12.576243 (+ 7509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:12.576253 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:12.576260 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:12.576309 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:12.580072 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:12.580856 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:12.580861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:12.580866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:12.580990 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:12.581439 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:12.581444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:12.581446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:12.581753 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:12.581894 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:12.582600 (+ 706us) server_negotiation.cc:299] Negotiation successful
0928 02:18:12.582790 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21644,"thread_start_us":22064,"threads_started":1}
I20260928 02:18:12.583343 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:12.539889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:60665 (local address 127.4.140.254:36419)
0928 02:18:12.559946 (+ 20057us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:12.559952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:12.559970 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:18:12.560033 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:12.560037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:12.560120 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:12.560222 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:12.563159 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:12.564378 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:12.567776 (+ 3398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:12.568483 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:12.568588 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:12.575567 (+ 6979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:12.575574 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:12.575579 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:12.575620 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:12.579541 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:12.580834 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:12.580839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:12.580843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:12.580969 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:12.581370 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:12.581375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:12.581378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:12.581639 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:12.581781 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:12.582560 (+ 779us) server_negotiation.cc:299] Negotiation successful
0928 02:18:12.582769 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19851,"thread_start_us":19834,"threads_started":1}
I20260928 02:18:12.584879 11777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:12.539998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.193:60665)
0928 02:18:12.558126 (+ 18128us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:12.558153 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:12.558322 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:12.560717 (+ 2395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:12.560721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:12.561546 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:12.562916 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:12.562947 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:12.564531 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:12.564536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:12.567604 (+ 3068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:12.567618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:12.568923 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:12.568929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:12.569365 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:12.570371 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:12.570410 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:12.575235 (+ 4825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:12.580020 (+ 4785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:12.580026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:12.580030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:12.580725 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:12.581085 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:12.581089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:12.581092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:12.581263 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:12.581759 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:12.581764 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:12.581920 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 02:18:12.582314 (+ 394us) client_negotiation.cc:240] Negotiation successful
0928 02:18:12.582874 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17882,"mutex_wait_us":638,"spinlock_wait_cycles":7936,"thread_start_us":17627,"threads_started":1}
I20260928 02:18:12.584916 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:12.539720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:45809 (local address 127.4.140.193:50727)
0928 02:18:12.551869 (+ 12149us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:12.551970 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:12.555330 (+ 3360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:12.561381 (+ 6051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:12.561388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:12.562016 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:12.562916 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:12.562974 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:12.564672 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:12.564678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:12.567604 (+ 2926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:12.567618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:12.568995 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:12.569000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:12.569378 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:12.570425 (+ 1047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:12.570448 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:12.575991 (+ 5543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:12.580176 (+ 4185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:12.580181 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:12.580184 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:12.580722 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:12.581133 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:12.581138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:12.581141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:12.581318 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:12.581964 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:12.581968 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:12.582148 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 02:18:12.582315 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 02:18:12.583785 (+ 1470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11674,"mutex_wait_us":1550,"spinlock_wait_cycles":6272,"thread_start_us":19028,"threads_started":1}
I20260928 02:18:12.584950 11775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:12.539720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:45985 (local address 127.4.140.193:56093)
0928 02:18:12.559291 (+ 19571us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:12.559313 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:12.559483 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:12.560296 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:12.560309 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:12.561546 (+ 1237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:12.562916 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:12.562947 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:12.564698 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:12.564702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:12.567582 (+ 2880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:12.567597 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:12.568922 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:12.568930 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:12.569375 (+ 445us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:12.570309 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:12.570358 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:12.574488 (+ 4130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:12.579240 (+ 4752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:12.579250 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:12.579270 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:12.579939 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:12.580417 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:12.580421 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:12.580425 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:12.580678 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:12.581426 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:12.581434 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:12.581919 (+ 485us) client_negotiation.cc:769] Sending connection context
0928 02:18:12.583304 (+ 1385us) client_negotiation.cc:240] Negotiation successful
0928 02:18:12.583600 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19422,"mutex_wait_us":1208,"spinlock_wait_cycles":45824,"thread_start_us":15504,"threads_started":1}
I20260928 02:18:12.586431 11770 heartbeater.cc:347] Connected to a master server at 127.4.140.254:36419
I20260928 02:18:12.586477 11769 heartbeater.cc:347] Connected to a master server at 127.4.140.252:45985
I20260928 02:18:12.586503 11771 heartbeater.cc:347] Connected to a master server at 127.4.140.253:45809
I20260928 02:18:12.586776 11769 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:12.586778 11771 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:12.586781 11770 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:12.587824 11769 heartbeater.cc:511] Master 127.4.140.252:45985 requested a full tablet report, sending...
I20260928 02:18:12.587863 11771 heartbeater.cc:511] Master 127.4.140.253:45809 requested a full tablet report, sending...
I20260928 02:18:12.587913 11770 heartbeater.cc:511] Master 127.4.140.254:36419 requested a full tablet report, sending...
I20260928 02:18:12.588150 11779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:12.540605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.4.140.194:45109)
0928 02:18:12.557408 (+ 16803us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:12.557422 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:12.559240 (+ 1818us) server_negotiation.cc:407] Connection header received
0928 02:18:12.559327 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:12.559339 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:12.559697 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:12.559851 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:12.563213 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:12.564766 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:12.566671 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:12.567565 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:12.567678 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:12.575708 (+ 8030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:12.575729 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:12.575751 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:12.575801 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:12.579853 (+ 4052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:12.582177 (+ 2324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:12.582186 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:12.582189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:12.582353 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:12.583934 (+ 1581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:12.583941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:12.583945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:12.584351 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:12.584652 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:12.586244 (+ 1592us) server_negotiation.cc:299] Negotiation successful
0928 02:18:12.587553 (+ 1309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16588,"thread_start_us":16426,"threads_started":1}
I20260928 02:18:12.589936 10797 ts_manager.cc:194] Re-registered known tserver with Master: 1d4a4eb5099e4262a21d9704f43f0bf1 (127.4.140.193:35389)
I20260928 02:18:12.590260 11402 ts_manager.cc:194] Registered new tserver with Master: 1d4a4eb5099e4262a21d9704f43f0bf1 (127.4.140.193:35389)
I20260928 02:18:12.590273 11589 ts_manager.cc:194] Registered new tserver with Master: 1d4a4eb5099e4262a21d9704f43f0bf1 (127.4.140.193:35389)
I20260928 02:18:12.591032 10797 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:60665
I20260928 02:18:12.591763 11116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:12.674908 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 10993
I20260928 02:18:12.691612 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:45109
--local_ip_for_outbound_sockets=127.4.140.194
--tserver_master_addrs=127.4.140.254:36419
--webserver_port=34875
--webserver_interface=127.4.140.194
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.194.keytab
--principal=oryx/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.140.254:36419,127.4.140.253:45809
--tserver_master_addrs=127.4.140.254:36419,127.4.140.253:45809,127.4.140.252:45985 with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:18:12.892688 11538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: NON_VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: true } } }
I20260928 02:18:12.893016 11538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:12.895140 11782 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197: attempting to promote NON_VOTER e8106b15bf894b04b0413a7a07fc5198 to VOTER
I20260928 02:18:12.895789 11782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: false } }
I20260928 02:18:12.897990 11412 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 1 FOLLOWER]: Refusing update from remote peer 4858ab829e994bd4a9c26227f4e81197: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 02:18:12.898427 11600 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 1 LEARNER]: Refusing update from remote peer 4858ab829e994bd4a9c26227f4e81197: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 02:18:12.898614 11538 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 02:18:12.899183 11783 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 02:18:12.901669 11538 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e8106b15bf894b04b0413a7a07fc5198 (127.4.140.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: false } } }
I20260928 02:18:12.904381 11600 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e8106b15bf894b04b0413a7a07fc5198 (127.4.140.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: false } } }
I20260928 02:18:12.905062 11782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: false } } }
I20260928 02:18:12.905298 11782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:12.905134 11783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: false } } }
I20260928 02:18:12.905434 11783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:12.905666 11538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: false } } }
I20260928 02:18:12.905876 11538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:12.905894 11412 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e8106b15bf894b04b0413a7a07fc5198 (127.4.140.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: false } } }
I20260928 02:18:12.909152 11788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: false } } }
I20260928 02:18:12.909549 11788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:18:12.910081 11789 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4858ab829e994bd4a9c26227f4e81197" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: false } } }
I20260928 02:18:12.910490 11789 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:18:13.028921 11781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:13.029256 11781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:13.029422 11781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:13.037019 11781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:13.037168 11781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:13.037330 11781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:18:13.048296 11781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.194.keytab
--principal=oryx/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:45109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=34875
--tserver_master_addrs=127.4.140.254:36419,127.4.140.253:45809,127.4.140.252:45985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:13.049878 11781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:13.051548 11781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:13.067473 11794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:13.067489 11795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:13.068233 11797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:13.068506 11781 server_base.cc:1034] running on GCE node
I20260928 02:18:13.069183 11781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:13.070183 11781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:13.071412 11781 hybrid_clock.cc:648] HybridClock initialized: now 1790561893071350 us; error 52 us; skew 500 ppm
I20260928 02:18:13.071834 11781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:13 dist-test-slave-h37r krb5kdc[10753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561893, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:13.075186 11781 init.cc:377] Logged in from keytab as oryx/127.4.140.194@KRBTEST.COM (short username oryx)
I20260928 02:18:13.076601 11781 webserver.cc:460] Webserver started at http://127.4.140.194:34875/ using document root <none> and password file <none>
I20260928 02:18:13.077286 11781 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:13.077485 11781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:13.082918 11781 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:18:13.085810 11804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:13.087193 11781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:18:13.087356 11781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "f1736eee0e104b78ad42edc586263f47"
format_stamp: "Formatted at 2026-09-28 02:18:09 on dist-test-slave-h37r"
server_key: "10afb852b7e678375928854286168f5c"
server_key_iv: "e013b143eb55c8c00cc31528ef9aa512"
server_key_version: "encryptionkey@0"
I20260928 02:18:13.088368 11781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:13.102109 11781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:13.106120 11781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:13.106391 11781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:13.107334 11781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:13.108757 11781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:13.108879 11781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:13.108999 11781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:13.109082 11781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:13.141423 11781 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:45109
I20260928 02:18:13.141469 11916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:45109 every 8 connection(s)
I20260928 02:18:13.142863 11781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:18:13.152343 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 11781
I20260928 02:18:13.152653 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 11126
Sep 28 02:18:13 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.140.195@KRBTEST.COM
Sep 28 02:18:13 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561893, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM
Sep 28 02:18:13 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561893, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.253@KRBTEST.COM
Sep 28 02:18:13 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561893, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.252@KRBTEST.COM
I20260928 02:18:13.203243 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.153720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51622 (local address 127.4.140.195:46077)
0928 02:18:13.170286 (+ 16566us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:13.170301 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:13.170352 (+ 51us) server_negotiation.cc:407] Connection header received
0928 02:18:13.170467 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:13.170480 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:13.170868 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:13.174842 (+ 3974us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:13.175213 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.177446 (+ 2233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.181573 (+ 4127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.182827 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.186383 (+ 3556us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.192134 (+ 5751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:13.192154 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:13.192175 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:13.192224 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:13.196079 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:13.196947 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:13.196956 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:13.196960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:13.197132 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:13.197599 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:13.197603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:13.197606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:13.198676 (+ 1070us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:13.201823 (+ 3147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:13.201995 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 02:18:13.202265 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16302,"thread_start_us":17444,"threads_started":1}
I20260928 02:18:13.205536 11249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:13.211336 11924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.146416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:45985 (local address 127.4.140.194:42269)
0928 02:18:13.184546 (+ 38130us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:13.184627 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:13.187616 (+ 2989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:13.188936 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:13.188950 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:13.191173 (+ 2223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:13.191443 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:13.191454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.193038 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.193044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:13.195699 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:13.195707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.196999 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.197006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:13.197446 (+ 440us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.198416 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:13.198441 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:13.202325 (+ 3884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:13.206106 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:13.206114 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:13.206117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:13.206696 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:13.207404 (+ 708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:13.207409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:13.207416 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:13.207655 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:13.208452 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:13.208460 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:13.208914 (+ 454us) client_negotiation.cc:769] Sending connection context
0928 02:18:13.209231 (+ 317us) client_negotiation.cc:240] Negotiation successful
0928 02:18:13.209848 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37827,"mutex_wait_us":421,"spinlock_wait_cycles":32768,"thread_start_us":41831,"threads_started":1}
I20260928 02:18:13.212046 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.147276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:44847 (local address 127.4.140.253:45809)
0928 02:18:13.183167 (+ 35891us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:13.183177 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:13.186639 (+ 3462us) server_negotiation.cc:407] Connection header received
0928 02:18:13.188165 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:13.188173 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:13.188296 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:13.188414 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:13.191649 (+ 3235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.192860 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.195855 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.196534 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.196633 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.201859 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:13.201865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:13.201871 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:13.201908 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:13.207684 (+ 5776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:13.208478 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:13.208483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:13.208486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:13.208621 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:13.209045 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:13.209049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:13.209052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:13.209406 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:13.209529 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:13.211432 (+ 1903us) server_negotiation.cc:299] Negotiation successful
0928 02:18:13.211652 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35692,"thread_start_us":36441,"threads_started":1}
I20260928 02:18:13.213037 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.147622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:42269 (local address 127.4.140.252:45985)
0928 02:18:13.183618 (+ 35996us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:13.183628 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:13.187985 (+ 4357us) server_negotiation.cc:407] Connection header received
0928 02:18:13.188076 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:13.188082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:13.188181 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:13.188283 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:13.191616 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.192860 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.195856 (+ 2996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.196534 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.196638 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.202544 (+ 5906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:13.202550 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:13.202554 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:13.202596 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:13.205986 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:13.207114 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:13.207121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:13.207127 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:13.207267 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:13.207782 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:13.207790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:13.207796 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:13.208296 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:13.208465 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:13.212644 (+ 4179us) server_negotiation.cc:299] Negotiation successful
0928 02:18:13.212797 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35838,"thread_start_us":36002,"threads_started":1}
I20260928 02:18:13.213444 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.145802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:34241 (local address 127.4.140.254:36419)
0928 02:18:13.190271 (+ 44469us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:13.190277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:13.190792 (+ 515us) server_negotiation.cc:407] Connection header received
0928 02:18:13.191040 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:13.191048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:13.191155 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:13.191273 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:13.192575 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.193895 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.196074 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.196715 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.196849 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.201844 (+ 4995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:13.201850 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:13.201855 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:13.201906 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:13.205764 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:13.207114 (+ 1350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:13.207123 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:13.207129 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:13.207313 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:13.208548 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:13.208556 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:13.208563 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:13.208977 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:13.209178 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:13.212560 (+ 3382us) server_negotiation.cc:299] Negotiation successful
0928 02:18:13.212890 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44328,"thread_start_us":44218,"threads_started":1}
I20260928 02:18:13.212064 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.146416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:45809 (local address 127.4.140.194:44847)
0928 02:18:13.185145 (+ 38729us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:13.185170 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:13.187629 (+ 2459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:13.188937 (+ 1308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:13.188951 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:13.190112 (+ 1161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:13.190945 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:13.190970 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.193038 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.193044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:13.195691 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:13.195706 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.196999 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.197006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:13.197436 (+ 430us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.198517 (+ 1081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:13.198544 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:13.201640 (+ 3096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:13.207814 (+ 6174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:13.207817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:13.207820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:13.208336 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:13.208742 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:13.208746 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:13.208749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:13.208914 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:13.209593 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:13.209677 (+ 84us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:13.209829 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 02:18:13.210034 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 02:18:13.210282 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37152,"mutex_wait_us":317,"spinlock_wait_cycles":2432,"thread_start_us":44406,"threads_started":1}
I20260928 02:18:13.214125 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.146448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.194:34241)
0928 02:18:13.190663 (+ 44215us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:13.190684 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:13.190822 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:13.191308 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:13.191315 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:13.191978 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:13.192300 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:13.192314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.194084 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.194090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:13.195861 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:13.195877 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.197105 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.197112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:13.197432 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.198177 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:13.198205 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:13.201544 (+ 3339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:13.205908 (+ 4364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:13.205920 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:13.205943 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:13.206980 (+ 1037us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:13.207794 (+ 814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:13.207800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:13.207803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:13.208053 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:13.211991 (+ 3938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:13.211997 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:13.212153 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 02:18:13.212296 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 02:18:13.212492 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44091,"mutex_wait_us":225,"thread_start_us":37290,"threads_started":1}
I20260928 02:18:13.214481 11917 heartbeater.cc:347] Connected to a master server at 127.4.140.252:45985
I20260928 02:18:13.214805 11917 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:13.215098 11918 heartbeater.cc:347] Connected to a master server at 127.4.140.254:36419
I20260928 02:18:13.215229 11918 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:13.215823 11918 heartbeater.cc:511] Master 127.4.140.254:36419 requested a full tablet report, sending...
I20260928 02:18:13.215845 11917 heartbeater.cc:511] Master 127.4.140.252:45985 requested a full tablet report, sending...
I20260928 02:18:13.216140 11919 heartbeater.cc:347] Connected to a master server at 127.4.140.253:45809
I20260928 02:18:13.216292 11919 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:13.216917 11919 heartbeater.cc:511] Master 127.4.140.253:45809 requested a full tablet report, sending...
I20260928 02:18:13.217715 11589 ts_manager.cc:194] Registered new tserver with Master: f1736eee0e104b78ad42edc586263f47 (127.4.140.194:45109)
I20260928 02:18:13.218220 10797 ts_manager.cc:194] Re-registered known tserver with Master: f1736eee0e104b78ad42edc586263f47 (127.4.140.194:45109)
I20260928 02:18:13.218434 11402 ts_manager.cc:194] Registered new tserver with Master: f1736eee0e104b78ad42edc586263f47 (127.4.140.194:45109)
I20260928 02:18:13.219347 10797 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:34241
I20260928 02:18:13.300714 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 11126
I20260928 02:18:13.317689 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:46077
--local_ip_for_outbound_sockets=127.4.140.195
--tserver_master_addrs=127.4.140.254:36419
--webserver_port=46811
--webserver_interface=127.4.140.195
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.195.keytab
--principal=oryx/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.140.254:36419,127.4.140.253:45809
--tserver_master_addrs=127.4.140.254:36419,127.4.140.253:45809,127.4.140.252:45985 with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:18:13.593480 11770 heartbeater.cc:503] Master 127.4.140.254:36419 was elected leader, sending a full tablet report...
W20260928 02:18:13.649500 11930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:13.649819 11930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:13.649974 11930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:13.657281 11930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:13.657424 11930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:13.657593 11930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:18:13.667927 11930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.195.keytab
--principal=oryx/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:46077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=46811
--tserver_master_addrs=127.4.140.254:36419,127.4.140.253:45809,127.4.140.252:45985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:13.669317 11930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:13.670707 11930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:13.684417 11936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:13.684376 11937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:13.684367 11939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:13.685024 11930 server_base.cc:1034] running on GCE node
I20260928 02:18:13.685734 11930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:13.686659 11930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:13.687877 11930 hybrid_clock.cc:648] HybridClock initialized: now 1790561893687819 us; error 60 us; skew 500 ppm
I20260928 02:18:13.688252 11930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:13 dist-test-slave-h37r krb5kdc[10753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561893, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:13.691269 11930 init.cc:377] Logged in from keytab as oryx/127.4.140.195@KRBTEST.COM (short username oryx)
I20260928 02:18:13.692850 11930 webserver.cc:460] Webserver started at http://127.4.140.195:46811/ using document root <none> and password file <none>
I20260928 02:18:13.693619 11930 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:13.693769 11930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:13.699259 11930 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:18:13.702050 11946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:13.703480 11930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:18:13.703620 11930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "c40dc0d1319b43d496ca89db9ca683c4"
format_stamp: "Formatted at 2026-09-28 02:18:10 on dist-test-slave-h37r"
server_key: "bbdbb19c5f7784df5f4fdb4f59abcf45"
server_key_iv: "d54fce7993954ded5c477d1b2ea810cd"
server_key_version: "encryptionkey@0"
I20260928 02:18:13.704687 11930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:13.727380 11930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:13.731756 11930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:13.732049 11930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:13.732998 11930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:13.734321 11930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:13.734408 11930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:13.734485 11930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:13.734541 11930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:13.765229 11930 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:46077
I20260928 02:18:13.765311 12058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:46077 every 8 connection(s)
I20260928 02:18:13.766646 11930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:18:13.767211 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 11930
Sep 28 02:18:13 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561893, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM
Sep 28 02:18:13 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561893, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.252@KRBTEST.COM
I20260928 02:18:13.829166 12064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.769826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:59033 (local address 127.4.140.254:36419)
0928 02:18:13.800504 (+ 30678us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:13.800513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:13.800538 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:18:13.807335 (+ 6797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:13.807342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:13.807470 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:13.807592 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:13.813269 (+ 5677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.814079 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.816224 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.816878 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.817010 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.821459 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:13.821466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:13.821470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:13.821514 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:13.825093 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:13.826100 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:13.826105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:13.826108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:13.826269 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:13.826775 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:13.826780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:13.826784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:13.827107 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:13.827245 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:13.828249 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0928 02:18:13.828599 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":30501,"thread_start_us":37172,"threads_started":1}
I20260928 02:18:13.830530 12068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.770496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.195:59033)
0928 02:18:13.794046 (+ 23550us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:13.794114 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:13.807142 (+ 13028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:13.807831 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:13.807838 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:13.808542 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:13.808879 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:13.808894 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.814231 (+ 5337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.814237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:13.816052 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:13.816074 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.817199 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.817207 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:13.817602 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.818693 (+ 1091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:13.818777 (+ 84us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:13.821198 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:13.825258 (+ 4060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:13.825269 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:13.825304 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:13.825951 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:13.826368 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:13.826373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:13.826378 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:13.826667 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:13.827227 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:13.827236 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:13.827709 (+ 473us) client_negotiation.cc:769] Sending connection context
0928 02:18:13.828070 (+ 361us) client_negotiation.cc:240] Negotiation successful
0928 02:18:13.828637 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23166,"thread_start_us":27014,"threads_started":1}
I20260928 02:18:13.832187 12060 heartbeater.cc:347] Connected to a master server at 127.4.140.254:36419
I20260928 02:18:13.832595 12060 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:13.833698 12060 heartbeater.cc:511] Master 127.4.140.254:36419 requested a full tablet report, sending...
I20260928 02:18:13.835775 10797 ts_manager.cc:194] Re-registered known tserver with Master: c40dc0d1319b43d496ca89db9ca683c4 (127.4.140.195:46077)
I20260928 02:18:13.837188 10797 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:59033
I20260928 02:18:13.842144 12065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.770124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:51425 (local address 127.4.140.252:45985)
0928 02:18:13.799870 (+ 29746us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:13.799880 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:13.799950 (+ 70us) server_negotiation.cc:407] Connection header received
0928 02:18:13.803369 (+ 3419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:13.803379 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:13.803528 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:13.803652 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:13.807068 (+ 3416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.808600 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.812320 (+ 3720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.812892 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.813057 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.834547 (+ 21490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:13.834552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:13.834556 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:13.834589 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:13.838465 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:13.839397 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:13.839403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:13.839408 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:13.839556 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:13.840051 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:13.840057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:13.840062 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:13.840429 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:13.840848 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:13.841397 (+ 549us) server_negotiation.cc:299] Negotiation successful
0928 02:18:13.841637 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29579,"thread_start_us":29849,"threads_started":1}
Sep 28 02:18:13 dist-test-slave-h37r krb5kdc[10753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561893, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.253@KRBTEST.COM
I20260928 02:18:13.844193 12069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.771175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:45985 (local address 127.4.140.195:51425)
0928 02:18:13.796481 (+ 25306us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:13.796503 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:13.802491 (+ 5988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:13.804289 (+ 1798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:13.804309 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:13.805561 (+ 1252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:13.806771 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:13.806808 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.809108 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.809116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:13.812114 (+ 2998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:13.812137 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.821472 (+ 9335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.821478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:13.821665 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.822704 (+ 1039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:13.822731 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:13.834348 (+ 11617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:13.838632 (+ 4284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:13.838638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:13.838644 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:13.839243 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:13.839716 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:13.839722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:13.839728 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:13.839884 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:13.840964 (+ 1080us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:13.840972 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:13.841125 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 02:18:13.841264 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 02:18:13.843666 (+ 2402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25095,"thread_start_us":22339,"threads_started":1}
I20260928 02:18:13.845098 12059 heartbeater.cc:347] Connected to a master server at 127.4.140.252:45985
I20260928 02:18:13.845240 12059 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:13.845710 12059 heartbeater.cc:511] Master 127.4.140.252:45985 requested a full tablet report, sending...
I20260928 02:18:13.846948 11589 ts_manager.cc:194] Registered new tserver with Master: c40dc0d1319b43d496ca89db9ca683c4 (127.4.140.195:46077)
I20260928 02:18:13.850556 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.815090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53314 (local address 127.4.140.253:45809)
0928 02:18:13.827343 (+ 12253us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:13.827349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:13.827364 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:18:13.827422 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:13.827427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:13.827501 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:13.830673 (+ 3172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:13.830894 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.832640 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.835886 (+ 3246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.836615 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.837905 (+ 1290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.840873 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:13.840880 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:13.840884 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:13.840922 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:13.845723 (+ 4801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:13.847616 (+ 1893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:13.847622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:13.847625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:13.847783 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:13.848312 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:13.848317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:13.848320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:13.848670 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:13.849623 (+ 953us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:13.849772 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 02:18:13.850078 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11567,"thread_start_us":11427,"threads_started":1}
I20260928 02:18:13.853686 12067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.771606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:45809 (local address 127.4.140.195:35179)
0928 02:18:13.797962 (+ 26356us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:13.797987 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:13.803187 (+ 5200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:13.813212 (+ 10025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:13.813217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:13.813789 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:13.814079 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:13.814092 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.823130 (+ 9038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.823140 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:13.825431 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:13.825465 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.834942 (+ 9477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.834952 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:13.837845 (+ 2893us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.841603 (+ 3758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:13.841624 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:13.844063 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:13.850652 (+ 6589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:13.850657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:13.850661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:13.851212 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:13.851668 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:13.851673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:13.851677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:13.851835 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:13.852580 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:13.852586 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:13.852765 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 02:18:13.852913 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 02:18:13.853251 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26202,"thread_start_us":24461,"threads_started":1}
I20260928 02:18:13.853730 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.770174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:35179 (local address 127.4.140.253:45809)
0928 02:18:13.812785 (+ 42611us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:13.812794 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:13.812818 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:18:13.812884 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:13.812892 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:13.813014 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:13.814244 (+ 1230us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:13.814397 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.822419 (+ 8022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.826181 (+ 3762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.833311 (+ 7130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.833424 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.845354 (+ 11930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:13.845364 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:13.845369 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:13.845410 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:13.850484 (+ 5074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:13.851365 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:13.851373 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:13.851378 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:13.851530 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:13.852011 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:13.852019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:13.852025 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:13.852377 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:13.852506 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:13.853038 (+ 532us) server_negotiation.cc:299] Negotiation successful
0928 02:18:13.853253 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42474,"thread_start_us":42276,"threads_started":1}
I20260928 02:18:13.855611 12061 heartbeater.cc:347] Connected to a master server at 127.4.140.253:45809
I20260928 02:18:13.855746 12061 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:13.856153 12061 heartbeater.cc:511] Master 127.4.140.253:45809 requested a full tablet report, sending...
I20260928 02:18:13.856560 12073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.814807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.4.140.254:36419)
0928 02:18:13.828000 (+ 13193us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:13.828008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:13.828027 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:18:13.828072 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:13.828076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:13.828138 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:13.828226 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:13.835460 (+ 7234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.841358 (+ 5898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.844374 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.844957 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.846229 (+ 1272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.850075 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:13.850082 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:13.850087 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:13.850129 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:13.853815 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:13.854530 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:13.854535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:13.854539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:13.854672 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:13.855053 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:13.855058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:13.855061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:13.855315 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:13.855556 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:13.855882 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 02:18:13.856153 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13055,"thread_start_us":12924,"threads_started":1}
I20260928 02:18:13.857514 11402 ts_manager.cc:194] Registered new tserver with Master: c40dc0d1319b43d496ca89db9ca683c4 (127.4.140.195:46077)
I20260928 02:18:13.859225 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.818379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52210 (local address 127.4.140.252:45985)
0928 02:18:13.818857 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:13.818863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:13.837262 (+ 18399us) server_negotiation.cc:407] Connection header received
0928 02:18:13.838340 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:13.838346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:13.838440 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:13.838523 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:13.844058 (+ 5535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.845136 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.847324 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.848073 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.848160 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.851801 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:13.851807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:13.851811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:13.851847 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:13.855425 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:13.856344 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:13.856351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:13.856356 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:13.856596 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:13.857213 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:13.857218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:13.857220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:13.857546 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:13.857779 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:13.858324 (+ 545us) server_negotiation.cc:299] Negotiation successful
0928 02:18:13.858826 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":1431,"threads_started":1}
I20260928 02:18:13.860787 10797 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:18:13.868227 10797 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:18:13.922996 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.891442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:35389 (local address 127.0.0.1:53286)
0928 02:18:13.913872 (+ 22430us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:13.913895 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:13.914061 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:13.915479 (+ 1418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:13.915484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:13.915511 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:13.915982 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:13.916001 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.918114 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.918119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:13.920183 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:13.920195 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.922216 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.922226 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:13.922388 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.922421 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:18:13.922498 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:18:13.922641 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22262,"thread_start_us":825,"threads_started":1}
I20260928 02:18:13.923738 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.892070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.4.140.193:35389)
0928 02:18:13.914732 (+ 22662us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:13.914741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:13.914774 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:18:13.914893 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:13.914904 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:13.915261 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:13.915497 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:13.916221 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.917955 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.920331 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.921696 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.921820 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.922065 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:13.923300 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0928 02:18:13.923471 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22471,"thread_start_us":23622,"threads_started":1}
I20260928 02:18:13.925925 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.892638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:45109 (local address 127.0.0.1:56710)
0928 02:18:13.917798 (+ 25160us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:13.917818 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:13.917943 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:13.918613 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:13.918618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:13.918652 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:13.919074 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:13.919083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.920902 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.920908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:13.923058 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:13.923075 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.925140 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.925147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:13.925330 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.925367 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:18:13.925478 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 02:18:13.925599 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25018,"thread_start_us":21516,"threads_started":1}
I20260928 02:18:13.926501 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.893190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.4.140.194:45109)
0928 02:18:13.912142 (+ 18952us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:13.912158 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:13.918143 (+ 5985us) server_negotiation.cc:407] Connection header received
0928 02:18:13.918229 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:13.918234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:13.918419 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:13.918628 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:13.919239 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.920749 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.923243 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.924652 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.924783 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.924984 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:13.926171 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0928 02:18:13.926316 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18675,"thread_start_us":18997,"threads_started":1}
I20260928 02:18:13.927978 11704 tablet_service.cc:1467] Processing CreateTablet for tablet a63816b469534bcc81fb912bd9cf7353 (DEFAULT_TABLE table=test-table [id=bfa40914ba654c7fb23458c35ea9db34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:13.929533 11704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63816b469534bcc81fb912bd9cf7353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:13.933100 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.891329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:46077 (local address 127.0.0.1:51624)
0928 02:18:13.892594 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:13.892615 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:13.892741 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:13.917103 (+ 24362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:13.917111 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:13.917154 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:13.917719 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:13.917729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.919725 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.919731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:13.922085 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:13.922100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.932215 (+ 10115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.932223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:13.932449 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.932486 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:18:13.932583 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 02:18:13.932732 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"thread_start_us":26109,"threads_started":1}
I20260928 02:18:13.934804 11852 tablet_service.cc:1467] Processing CreateTablet for tablet a63816b469534bcc81fb912bd9cf7353 (DEFAULT_TABLE table=test-table [id=bfa40914ba654c7fb23458c35ea9db34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:13.935853 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:13.891670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51624 (local address 127.4.140.195:46077)
0928 02:18:13.916428 (+ 24758us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:13.916437 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:13.916475 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:18:13.916572 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:13.916581 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:13.916854 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:13.917072 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:13.917910 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.919579 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.923986 (+ 4407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:13.925429 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:13.925538 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:13.925732 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:13.932670 (+ 6938us) server_negotiation.cc:299] Negotiation successful
0928 02:18:13.935516 (+ 2846us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24561,"thread_start_us":24378,"threads_started":1}
I20260928 02:18:13.936085 11852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63816b469534bcc81fb912bd9cf7353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:13.939831 11994 tablet_service.cc:1467] Processing CreateTablet for tablet a63816b469534bcc81fb912bd9cf7353 (DEFAULT_TABLE table=test-table [id=bfa40914ba654c7fb23458c35ea9db34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:13.941357 11994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63816b469534bcc81fb912bd9cf7353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:13.947374 12093 tablet_bootstrap.cc:492] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1: Bootstrap starting.
I20260928 02:18:13.950613 12093 tablet_bootstrap.cc:654] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:13.951713 12093 log.cc:824] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:13.954039 12095 tablet_bootstrap.cc:492] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47: Bootstrap starting.
I20260928 02:18:13.954878 12093 tablet_bootstrap.cc:492] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1: No bootstrap required, opened a new log
I20260928 02:18:13.955186 12093 ts_tablet_manager.cc:1397] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260928 02:18:13.956560 12096 tablet_bootstrap.cc:492] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4: Bootstrap starting.
I20260928 02:18:13.957897 12095 tablet_bootstrap.cc:654] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:13.959201 12095 log.cc:824] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:13.959473 12096 tablet_bootstrap.cc:654] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:13.960736 12096 log.cc:824] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:13.963320 12095 tablet_bootstrap.cc:492] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47: No bootstrap required, opened a new log
I20260928 02:18:13.963624 12095 ts_tablet_manager.cc:1397] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260928 02:18:13.966831 12093 raft_consensus.cc:348] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 } } peers { permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 } } peers { permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 } }
I20260928 02:18:13.967442 12093 raft_consensus.cc:374] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:13.967577 12093 raft_consensus.cc:729] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4a4eb5099e4262a21d9704f43f0bf1, State: Initialized, Role: FOLLOWER
I20260928 02:18:13.968372 12093 consensus_queue.cc:260] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 } } peers { permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 } } peers { permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 } }
I20260928 02:18:13.970264 12096 tablet_bootstrap.cc:492] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4: No bootstrap required, opened a new log
I20260928 02:18:13.970584 12096 ts_tablet_manager.cc:1397] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260928 02:18:13.974040 12093 ts_tablet_manager.cc:1428] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1: Time spent starting tablet: real 0.019s user 0.011s sys 0.006s
I20260928 02:18:13.979481 12095 raft_consensus.cc:348] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 } } peers { permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 } } peers { permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 } }
I20260928 02:18:13.979902 12095 raft_consensus.cc:374] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:13.980058 12095 raft_consensus.cc:729] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1736eee0e104b78ad42edc586263f47, State: Initialized, Role: FOLLOWER
I20260928 02:18:13.980477 12096 raft_consensus.cc:348] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 } } peers { permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 } } peers { permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 } }
I20260928 02:18:13.980849 12096 raft_consensus.cc:374] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:13.980954 12096 raft_consensus.cc:729] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c40dc0d1319b43d496ca89db9ca683c4, State: Initialized, Role: FOLLOWER
I20260928 02:18:13.980783 12095 consensus_queue.cc:260] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 } } peers { permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 } } peers { permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 } }
I20260928 02:18:13.981546 12096 consensus_queue.cc:260] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 } } peers { permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 } } peers { permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 } }
I20260928 02:18:13.983428 12060 heartbeater.cc:503] Master 127.4.140.254:36419 was elected leader, sending a full tablet report...
I20260928 02:18:13.983757 12095 ts_tablet_manager.cc:1428] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47: Time spent starting tablet: real 0.020s user 0.010s sys 0.006s
I20260928 02:18:13.984700 12096 ts_tablet_manager.cc:1428] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260928 02:18:13.985251 11918 heartbeater.cc:503] Master 127.4.140.254:36419 was elected leader, sending a full tablet report...
I20260928 02:18:14.039696 12101 raft_consensus.cc:482] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:18:14.040030 12101 raft_consensus.cc:504] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 } } peers { permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 } } peers { permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 } }
I20260928 02:18:14.042025 12101 leader_election.cc:288] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d4a4eb5099e4262a21d9704f43f0bf1 (127.4.140.193:35389), f1736eee0e104b78ad42edc586263f47 (127.4.140.194:45109)
W20260928 02:18:14.046417 11772 tablet.cc:2369] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:18:14.048197 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:14.042462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:51077 (local address 127.4.140.193:35389)
0928 02:18:14.042729 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:14.042736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:14.042757 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:18:14.042934 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:14.042941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:14.043021 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:14.043164 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:14.043610 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:14.044388 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:14.046094 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:14.047063 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:14.047208 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:14.047276 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:14.047785 (+ 509us) server_negotiation.cc:299] Negotiation successful
0928 02:18:14.047960 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260928 02:18:14.048195 12067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:14.042350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:35389 (local address 127.4.140.195:51077)
0928 02:18:14.042607 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:14.042629 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:14.042782 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:14.043188 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:14.043193 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:14.043216 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:14.043460 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:14.043472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:14.044526 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:14.044534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:14.045921 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:14.045932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:14.047386 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:14.047392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:14.047600 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:14.047648 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 02:18:14.047741 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 02:18:14.047868 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":24}
I20260928 02:18:14.048622 12069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:14.042729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:45109 (local address 127.4.140.195:44845)
0928 02:18:14.042916 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:14.042932 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:14.043021 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:14.043394 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:14.043397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:14.043419 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:14.043590 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:14.043600 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:14.044642 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:14.044645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:14.045940 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:14.045951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:14.047523 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:14.047528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:14.047783 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:14.047816 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:18:14.047941 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 02:18:14.048095 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":20}
I20260928 02:18:14.049132 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:14.042850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:44845 (local address 127.4.140.194:45109)
0928 02:18:14.043064 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:14.043068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:14.043084 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:18:14.043163 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:14.043168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:14.043257 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:14.043354 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:14.043738 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:14.044495 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:14.046127 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:14.047351 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:14.047515 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:14.047583 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:14.048753 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0928 02:18:14.048891 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:18:14.049206 11724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63816b469534bcc81fb912bd9cf7353" candidate_uuid: "c40dc0d1319b43d496ca89db9ca683c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" is_pre_election: true
I20260928 02:18:14.049746 11724 raft_consensus.cc:2454] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c40dc0d1319b43d496ca89db9ca683c4 in term 0.
I20260928 02:18:14.050019 11872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63816b469534bcc81fb912bd9cf7353" candidate_uuid: "c40dc0d1319b43d496ca89db9ca683c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1736eee0e104b78ad42edc586263f47" is_pre_election: true
I20260928 02:18:14.050419 11950 leader_election.cc:302] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4a4eb5099e4262a21d9704f43f0bf1, c40dc0d1319b43d496ca89db9ca683c4; no voters:
I20260928 02:18:14.050498 11872 raft_consensus.cc:2454] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c40dc0d1319b43d496ca89db9ca683c4 in term 0.
I20260928 02:18:14.050827 12101 raft_consensus.cc:2781] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:18:14.051036 12101 raft_consensus.cc:482] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:18:14.051198 12101 raft_consensus.cc:3037] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:14.053522 12101 raft_consensus.cc:504] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 } } peers { permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 } } peers { permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 } }
I20260928 02:18:14.054106 12101 leader_election.cc:288] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [CANDIDATE]: Term 1 election: Requested vote from peers 1d4a4eb5099e4262a21d9704f43f0bf1 (127.4.140.193:35389), f1736eee0e104b78ad42edc586263f47 (127.4.140.194:45109)
I20260928 02:18:14.054503 11724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63816b469534bcc81fb912bd9cf7353" candidate_uuid: "c40dc0d1319b43d496ca89db9ca683c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1"
I20260928 02:18:14.054637 11872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63816b469534bcc81fb912bd9cf7353" candidate_uuid: "c40dc0d1319b43d496ca89db9ca683c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1736eee0e104b78ad42edc586263f47"
I20260928 02:18:14.054700 11724 raft_consensus.cc:3037] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:14.054836 11872 raft_consensus.cc:3037] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47 [term 0 FOLLOWER]: Advancing to term 1
W20260928 02:18:14.055981 12063 tablet.cc:2369] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:18:14.057507 11724 raft_consensus.cc:2454] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c40dc0d1319b43d496ca89db9ca683c4 in term 1.
I20260928 02:18:14.057858 11950 leader_election.cc:302] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4a4eb5099e4262a21d9704f43f0bf1, c40dc0d1319b43d496ca89db9ca683c4; no voters:
I20260928 02:18:14.057952 11872 raft_consensus.cc:2454] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c40dc0d1319b43d496ca89db9ca683c4 in term 1.
I20260928 02:18:14.058192 12101 raft_consensus.cc:2781] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:18:14.058754 12101 raft_consensus.cc:686] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [term 1 LEADER]: Becoming Leader. State: Replica: c40dc0d1319b43d496ca89db9ca683c4, State: Running, Role: LEADER
I20260928 02:18:14.059350 12101 consensus_queue.cc:237] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 } } peers { permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 } } peers { permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 } }
I20260928 02:18:14.064621 10797 catalog_manager.cc:5495] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c40dc0d1319b43d496ca89db9ca683c4 (127.4.140.195). New cstate: current_term: 1 leader_uuid: "c40dc0d1319b43d496ca89db9ca683c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 } health_report { overall_health: HEALTHY } } }
I20260928 02:18:14.100876 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:14.095616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.4.140.195:46077)
0928 02:18:14.095838 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:14.095843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:14.095861 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:18:14.096031 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:14.096036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:14.096121 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:14.096262 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:18:14.096632 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:14.097338 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:14.098658 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:14.099149 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:14.099253 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:14.099679 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:18:14.100386 (+ 707us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:18:14.100505 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:14.100609 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 02:18:14.100675 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 02:18:14.109344 11872 raft_consensus.cc:1261] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47 [term 1 FOLLOWER]: Refusing update from remote peer c40dc0d1319b43d496ca89db9ca683c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:18:14.109603 11724 raft_consensus.cc:1261] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 1 FOLLOWER]: Refusing update from remote peer c40dc0d1319b43d496ca89db9ca683c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:18:14.110392 12102 consensus_queue.cc:1035] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:18:14.110792 12101 consensus_queue.cc:1035] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:18:14.128015 12105 mvcc.cc:205] Tried to move back new op lower bound from 7334141518261362688 to 7334141518071611392. Current Snapshot: MvccSnapshot[applied={T|T < 7334141518261362688}]
I20260928 02:18:14.152187 10797 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:18:14.152477 10797 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:18:14.160461 10797 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4858ab829e994bd4a9c26227f4e81197: Sending DeleteTablet for 3 replicas of tablet a63816b469534bcc81fb912bd9cf7353
I20260928 02:18:14.161583 11852 tablet_service.cc:1514] Processing DeleteTablet for tablet a63816b469534bcc81fb912bd9cf7353 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:14 UTC) from {username='oryx', principal='oryx/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:56710
I20260928 02:18:14.161890 11704 tablet_service.cc:1514] Processing DeleteTablet for tablet a63816b469534bcc81fb912bd9cf7353 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:14 UTC) from {username='oryx', principal='oryx/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:53286
I20260928 02:18:14.165346 11994 tablet_service.cc:1514] Processing DeleteTablet for tablet a63816b469534bcc81fb912bd9cf7353 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:14 UTC) from {username='oryx', principal='oryx/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:51624
I20260928 02:18:14.171623 12116 tablet_replica.cc:330] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1: stopping tablet replica
I20260928 02:18:14.172149 12116 raft_consensus.cc:2229] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:18:14.173203 12116 raft_consensus.cc:2258] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:14.174222 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 10766
I20260928 02:18:14.175556 10828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:14.175643 12116 ts_tablet_manager.cc:1905] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:14.181924 12118 tablet_replica.cc:330] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4: stopping tablet replica
I20260928 02:18:14.182518 12118 raft_consensus.cc:2229] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:18:14.183230 12118 raft_consensus.cc:2258] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:14.184358 12116 ts_tablet_manager.cc:1918] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:18:14.184525 12116 log.cc:1196] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/a63816b469534bcc81fb912bd9cf7353
I20260928 02:18:14.185091 12116 ts_tablet_manager.cc:1939] T a63816b469534bcc81fb912bd9cf7353 P 1d4a4eb5099e4262a21d9704f43f0bf1: Deleting consensus metadata
I20260928 02:18:14.185993 12118 ts_tablet_manager.cc:1905] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:14.194208 12118 ts_tablet_manager.cc:1918] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:18:14.194469 12115 tablet_replica.cc:330] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47: stopping tablet replica
I20260928 02:18:14.194494 12118 log.cc:1196] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/a63816b469534bcc81fb912bd9cf7353
I20260928 02:18:14.194962 12115 raft_consensus.cc:2229] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:18:14.195261 12118 ts_tablet_manager.cc:1939] T a63816b469534bcc81fb912bd9cf7353 P c40dc0d1319b43d496ca89db9ca683c4: Deleting consensus metadata
I20260928 02:18:14.195528 12115 raft_consensus.cc:2258] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:14.197822 12115 ts_tablet_manager.cc:1905] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:14.204171 12115 ts_tablet_manager.cc:1918] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:18:14.204326 12115 log.cc:1196] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/a63816b469534bcc81fb912bd9cf7353
I20260928 02:18:14.204862 12115 ts_tablet_manager.cc:1939] T a63816b469534bcc81fb912bd9cf7353 P f1736eee0e104b78ad42edc586263f47: Deleting consensus metadata
I20260928 02:18:14.206173 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:14.164368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.4.140.253:45809)
0928 02:18:14.164578 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:14.164585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:14.173726 (+ 9141us) server_negotiation.cc:407] Connection header received
0928 02:18:14.174971 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:14.174978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:14.175096 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:14.175186 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:14.189308 (+ 14122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:14.190484 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:14.194081 (+ 3597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:14.194755 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:14.194857 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:14.199146 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:14.199155 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:14.199161 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:14.199205 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:14.202859 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:14.203754 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:14.203760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:14.203764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:14.203938 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:14.204386 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:14.204391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:14.204395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:14.204751 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:14.204968 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:14.205572 (+ 604us) server_negotiation.cc:299] Negotiation successful
0928 02:18:14.205833 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:18:14.223203 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:14.172931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.4.140.252:45985)
0928 02:18:14.173116 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:14.173122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:14.207148 (+ 34026us) server_negotiation.cc:407] Connection header received
0928 02:18:14.207396 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:14.207402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:14.207488 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:14.207558 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:14.209518 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:14.210539 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:14.212462 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:14.213196 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:14.213319 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:14.216660 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:14.216668 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:14.216675 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:14.216721 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:14.220394 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:14.221176 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:14.221181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:14.221186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:14.221339 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:14.221740 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:14.221745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:14.221748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:14.222075 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:14.222323 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:14.222634 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 02:18:14.222857 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:18:14.319857 10782 catalog_manager.cc:4841] TS c40dc0d1319b43d496ca89db9ca683c4 (127.4.140.195:46077): tablet a63816b469534bcc81fb912bd9cf7353 (table test-table [id=bfa40914ba654c7fb23458c35ea9db34]) successfully deleted
I20260928 02:18:14.320309 10785 catalog_manager.cc:4841] TS 1d4a4eb5099e4262a21d9704f43f0bf1 (127.4.140.193:35389): tablet a63816b469534bcc81fb912bd9cf7353 (table test-table [id=bfa40914ba654c7fb23458c35ea9db34]) successfully deleted
W20260928 02:18:14.320505 10785 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.140.193:35389, user_credentials={real_user=slave}} blocked reactor thread for 133622us
I20260928 02:18:14.320680 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 10766
W20260928 02:18:14.349464 11806 connection.cc:535] server connection from 127.0.0.1:56710 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:18:14.372658 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:14.354766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53332 (local address 127.4.140.253:45809)
0928 02:18:14.354976 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:14.354982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:14.357466 (+ 2484us) server_negotiation.cc:407] Connection header received
0928 02:18:14.357802 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:14.357811 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:14.357916 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:14.358004 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:14.360461 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:14.361622 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:14.363006 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:14.363621 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:14.363746 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:14.366894 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:14.366899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:14.366902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:14.366942 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:14.369688 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:14.370268 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:14.370273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:14.370276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:14.370453 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:14.370825 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:14.370830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:14.370833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:14.371193 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:14.371592 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:14.372192 (+ 600us) server_negotiation.cc:299] Negotiation successful
0928 02:18:14.372411 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:18:14.375108 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:14.354362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52218 (local address 127.4.140.252:45985)
0928 02:18:14.357045 (+ 2683us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:14.357050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:14.358726 (+ 1676us) server_negotiation.cc:407] Connection header received
0928 02:18:14.359388 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:14.359395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:14.359477 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:14.359626 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:14.361587 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:14.362751 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:14.364436 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:14.365129 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:14.365312 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:14.368669 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:14.368676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:14.368680 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:14.368718 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:14.371994 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:14.372743 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:14.372747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:14.372750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:14.372894 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:14.373456 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:14.373463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:14.373466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:14.373861 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:14.374154 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:14.374521 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 02:18:14.374731 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2534}
I20260928 02:18:15.131992 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.131184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.194:40551)
0928 02:18:15.131643 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.131761 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":233,"threads_started":1}
W20260928 02:18:15.132449 11918 heartbeater.cc:644] Failed to heartbeat to 127.4.140.254:36419 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
I20260928 02:18:15.144636 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.143683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.193:49257)
0928 02:18:15.144276 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.144423 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":324,"threads_started":1}
W20260928 02:18:15.145030 11770 heartbeater.cc:644] Failed to heartbeat to 127.4.140.254:36419 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
I20260928 02:18:15.149343 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.148385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.195:41107)
0928 02:18:15.148944 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.149092 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":334,"threads_started":1}
W20260928 02:18:15.149820 12060 heartbeater.cc:644] Failed to heartbeat to 127.4.140.254:36419 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
I20260928 02:18:15.659848 12130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4858ab829e994bd4a9c26227f4e81197)
I20260928 02:18:15.660152 12130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: false } }
I20260928 02:18:15.660372 12131 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4858ab829e994bd4a9c26227f4e81197)
I20260928 02:18:15.660632 12131 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: false } }
I20260928 02:18:15.661118 12130 leader_election.cc:288] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4858ab829e994bd4a9c26227f4e81197 (127.4.140.254:36419), e8106b15bf894b04b0413a7a07fc5198 (127.4.140.252:45985)
I20260928 02:18:15.661572 12131 leader_election.cc:288] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4858ab829e994bd4a9c26227f4e81197 (127.4.140.254:36419), 1878aa81226947478f7f8470d8f0452f (127.4.140.253:45809)
I20260928 02:18:15.665799 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.661231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.0.0.1:37164)
0928 02:18:15.665221 (+ 3990us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.665444 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3880,"threads_started":1}
I20260928 02:18:15.669041 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.662018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.0.0.1:37168)
0928 02:18:15.668424 (+ 6406us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.668690 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6223,"thread_start_us":5197,"threads_started":1}
W20260928 02:18:15.669348 11576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
W20260928 02:18:15.674062 11389 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
I20260928 02:18:15.674403 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.673923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.0.0.1:37174)
0928 02:18:15.674124 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.674227 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260928 02:18:15.674634 11576 leader_election.cc:334] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4858ab829e994bd4a9c26227f4e81197 (127.4.140.254:36419): Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
I20260928 02:18:15.677182 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.676631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.0.0.1:37178)
0928 02:18:15.676792 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.676967 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 02:18:15.677394 11389 leader_election.cc:334] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4858ab829e994bd4a9c26227f4e81197 (127.4.140.254:36419): Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
I20260928 02:18:15.679534 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.661870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:45809 (local address 127.0.0.1:39052)
0928 02:18:15.666902 (+ 5032us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.666926 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:15.667084 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:15.674076 (+ 6992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:15.674081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:15.674109 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:15.674468 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:15.674482 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.675858 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.675866 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:15.677674 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:15.677690 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.678739 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.678745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:15.678946 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:15.678989 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:18:15.679095 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:18:15.679225 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4856,"thread_start_us":6190,"threads_started":1}
I20260928 02:18:15.679625 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.665221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.4.140.253:45809)
0928 02:18:15.673581 (+ 8360us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:15.673585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:15.673601 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:18:15.673651 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:15.673655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:15.673920 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:15.674674 (+ 754us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:15.674901 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.675725 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.677790 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.678546 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.678689 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:15.678782 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:15.679128 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 02:18:15.679307 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":8252,"thread_start_us":8034,"threads_started":1}
I20260928 02:18:15.681208 11412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8106b15bf894b04b0413a7a07fc5198" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1878aa81226947478f7f8470d8f0452f" is_pre_election: true
I20260928 02:18:15.681476 11412 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8106b15bf894b04b0413a7a07fc5198 in term 1.
I20260928 02:18:15.681980 11573 leader_election.cc:302] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1878aa81226947478f7f8470d8f0452f, e8106b15bf894b04b0413a7a07fc5198; no voters: 4858ab829e994bd4a9c26227f4e81197
I20260928 02:18:15.682339 12131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 02:18:15.682474 12131 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4858ab829e994bd4a9c26227f4e81197)
I20260928 02:18:15.682569 12131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:18:15.682789 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.661325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:45985 (local address 127.0.0.1:40232)
0928 02:18:15.667609 (+ 6284us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.667632 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:15.667766 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:15.677279 (+ 9513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:15.677284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:15.677325 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:15.677592 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:15.677606 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.678902 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.678907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:15.680627 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:15.680638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.681994 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.682003 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:15.682192 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:15.682230 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:18:15.682315 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 02:18:15.682459 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6145,"mutex_wait_us":2,"thread_start_us":3658,"threads_started":1}
I20260928 02:18:15.684732 12131 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: false } }
I20260928 02:18:15.685366 12131 leader_election.cc:288] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [CANDIDATE]: Term 2 election: Requested vote from peers 4858ab829e994bd4a9c26227f4e81197 (127.4.140.254:36419), 1878aa81226947478f7f8470d8f0452f (127.4.140.253:45809)
I20260928 02:18:15.685393 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.668380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40232 (local address 127.4.140.252:45985)
0928 02:18:15.676965 (+ 8585us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:15.676971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:15.676987 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:18:15.677050 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:15.677055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:15.677125 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:15.677629 (+ 504us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:15.677916 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.678760 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.680767 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.681831 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.681979 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:15.682099 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:15.684981 (+ 2882us) server_negotiation.cc:299] Negotiation successful
0928 02:18:15.685082 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8443,"thread_start_us":7992,"threads_started":1}
I20260928 02:18:15.685828 11412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8106b15bf894b04b0413a7a07fc5198" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1878aa81226947478f7f8470d8f0452f"
I20260928 02:18:15.685936 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.685506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.0.0.1:37180)
0928 02:18:15.685736 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.685813 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260928 02:18:15.686061 11412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:18:15.686637 11600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1878aa81226947478f7f8470d8f0452f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e8106b15bf894b04b0413a7a07fc5198" is_pre_election: true
I20260928 02:18:15.686995 11600 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1878aa81226947478f7f8470d8f0452f in current term 2: Already voted for candidate e8106b15bf894b04b0413a7a07fc5198 in this term.
I20260928 02:18:15.687019 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.686537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.0.0.1:37184)
0928 02:18:15.686685 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.686822 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 02:18:15.687213 11576 leader_election.cc:334] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4858ab829e994bd4a9c26227f4e81197 (127.4.140.254:36419): Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
I20260928 02:18:15.687625 11387 leader_election.cc:302] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1878aa81226947478f7f8470d8f0452f; no voters: 4858ab829e994bd4a9c26227f4e81197, e8106b15bf894b04b0413a7a07fc5198
I20260928 02:18:15.688661 11412 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8106b15bf894b04b0413a7a07fc5198 in term 2.
I20260928 02:18:15.688815 12130 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:18:15.689087 11573 leader_election.cc:302] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1878aa81226947478f7f8470d8f0452f, e8106b15bf894b04b0413a7a07fc5198; no voters: 4858ab829e994bd4a9c26227f4e81197
I20260928 02:18:15.689409 12131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:18:15.689815 12131 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [term 2 LEADER]: Becoming Leader. State: Replica: e8106b15bf894b04b0413a7a07fc5198, State: Running, Role: LEADER
I20260928 02:18:15.690171 12131 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: false } }
I20260928 02:18:15.692395 12140 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8106b15bf894b04b0413a7a07fc5198. Latest consensus state: current_term: 2 leader_uuid: "e8106b15bf894b04b0413a7a07fc5198" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: false } } }
I20260928 02:18:15.692667 12140 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:15.693275 12142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:18:15.696964 12142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:18:15.698078 12142 catalog_manager.cc:1259] Loaded cluster ID: dfd33f39829443b9ac381d5072a1c40f
I20260928 02:18:15.698148 12142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:18:15.699276 12142 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:18:15.700102 12142 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: Loaded TSK: 0
I20260928 02:18:15.700923 12142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:18:15.752349 11588 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:18:15.759289 11588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:18:15.763556 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.763173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.0.0.1:37204)
0928 02:18:15.763341 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.763414 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 02:18:15.763614 11412 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [term 2 FOLLOWER]: Refusing update from remote peer e8106b15bf894b04b0413a7a07fc5198: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 02:18:15.764334 12140 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 02:18:15.764557 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.764161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.0.0.1:37216)
0928 02:18:15.764327 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.764395 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 02:18:15.764773 11576 consensus_peers.cc:487] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 -> Peer 4858ab829e994bd4a9c26227f4e81197 (127.4.140.254:36419): Couldn't send request to peer 4858ab829e994bd4a9c26227f4e81197. Status: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:18:15.766916 12130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8106b15bf894b04b0413a7a07fc5198. Latest consensus state: current_term: 2 leader_uuid: "e8106b15bf894b04b0413a7a07fc5198" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: false } } }
I20260928 02:18:15.767230 12130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:18:15.768564 12143 mvcc.cc:205] Tried to move back new op lower bound from 7334141525041504256 to 7334141524750147584. Current Snapshot: MvccSnapshot[applied={T|T < 7334141525041504256}]
I20260928 02:18:15.769375 12140 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e8106b15bf894b04b0413a7a07fc5198" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: false } } }
I20260928 02:18:15.769651 12140 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:15.776397 12130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e8106b15bf894b04b0413a7a07fc5198" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4858ab829e994bd4a9c26227f4e81197" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36419 } } peers { permanent_uuid: "1878aa81226947478f7f8470d8f0452f" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 45809 } attrs { promote: false } } peers { permanent_uuid: "e8106b15bf894b04b0413a7a07fc5198" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 45985 } attrs { promote: false } } }
I20260928 02:18:15.776728 12130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1878aa81226947478f7f8470d8f0452f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:18:15.791503 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.781186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:46077 (local address 127.0.0.1:44082)
0928 02:18:15.781573 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.781596 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:15.781733 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:15.785440 (+ 3707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:15.785447 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:15.785477 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:15.785803 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:15.785811 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.786782 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.786791 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:15.788745 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:15.788768 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.790646 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.790652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:15.790855 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:15.790909 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 02:18:15.790974 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:18:15.791143 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":21}
I20260928 02:18:15.792184 12152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.781958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50906 (local address 127.4.140.193:35389)
0928 02:18:15.782506 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:15.782515 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:15.784556 (+ 2041us) server_negotiation.cc:407] Connection header received
0928 02:18:15.784835 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:15.784844 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:15.784921 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:15.785020 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:15.785765 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.787221 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.789607 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.790839 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.790937 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:15.791075 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:15.791687 (+ 612us) server_negotiation.cc:299] Negotiation successful
0928 02:18:15.791828 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":268,"threads_started":1}
I20260928 02:18:15.792405 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.781539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44082 (local address 127.4.140.195:46077)
0928 02:18:15.785057 (+ 3518us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:15.785067 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:15.785090 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:18:15.785173 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:15.785181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:15.785289 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:15.785412 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:15.785938 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.786656 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.789105 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.790400 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.791750 (+ 1350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:15.791850 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:15.791934 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 02:18:15.792074 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3392,"thread_start_us":3284,"threads_started":1}
I20260928 02:18:15.792989 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.781505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:35389 (local address 127.0.0.1:50906)
0928 02:18:15.784456 (+ 2951us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.784483 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:15.784643 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:15.785169 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:15.785173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:15.785190 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:15.785582 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:15.785600 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.787368 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.787373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:15.789426 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:15.789440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.791289 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.791295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:15.791477 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:15.791517 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:18:15.792500 (+ 983us) client_negotiation.cc:240] Negotiation successful
0928 02:18:15.792651 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2789,"mutex_wait_us":6,"thread_start_us":2573,"threads_started":1}
I20260928 02:18:15.793179 11994 tablet_service.cc:1467] Processing CreateTablet for tablet f8c76e8bc14e4a31a53e069b5d5b0b93 (DEFAULT_TABLE table=test-table [id=f5b39dc3b1924763927b386baaaee4f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:15.793547 11994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8c76e8bc14e4a31a53e069b5d5b0b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:15.796933 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.780716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:45109 (local address 127.0.0.1:47924)
0928 02:18:15.780943 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.780961 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:15.781066 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:15.788543 (+ 7477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:15.788547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:15.788571 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:15.788801 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:15.788809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.790106 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.790113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:15.791383 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:15.791395 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.796164 (+ 4769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.796174 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:15.796413 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:15.796460 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 02:18:15.796553 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 02:18:15.796684 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260928 02:18:15.797035 11704 tablet_service.cc:1467] Processing CreateTablet for tablet f8c76e8bc14e4a31a53e069b5d5b0b93 (DEFAULT_TABLE table=test-table [id=f5b39dc3b1924763927b386baaaee4f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:15.797330 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.780830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47924 (local address 127.4.140.194:45109)
0928 02:18:15.788276 (+ 7446us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:15.788280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:15.788297 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:18:15.788350 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:15.788354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:15.788407 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:15.788519 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:15.788953 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.789957 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.791523 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.792765 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.792898 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:15.792992 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:15.796861 (+ 3869us) server_negotiation.cc:299] Negotiation successful
0928 02:18:15.797009 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7350,"thread_start_us":7267,"threads_started":1}
I20260928 02:18:15.797482 11704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8c76e8bc14e4a31a53e069b5d5b0b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:15.798372 11852 tablet_service.cc:1467] Processing CreateTablet for tablet f8c76e8bc14e4a31a53e069b5d5b0b93 (DEFAULT_TABLE table=test-table [id=f5b39dc3b1924763927b386baaaee4f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:15.798779 11852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8c76e8bc14e4a31a53e069b5d5b0b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:15.798853 12153 tablet_bootstrap.cc:492] T f8c76e8bc14e4a31a53e069b5d5b0b93 P c40dc0d1319b43d496ca89db9ca683c4: Bootstrap starting.
I20260928 02:18:15.800318 12153 tablet_bootstrap.cc:654] T f8c76e8bc14e4a31a53e069b5d5b0b93 P c40dc0d1319b43d496ca89db9ca683c4: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:15.802282 12153 tablet_bootstrap.cc:492] T f8c76e8bc14e4a31a53e069b5d5b0b93 P c40dc0d1319b43d496ca89db9ca683c4: No bootstrap required, opened a new log
I20260928 02:18:15.802430 12153 ts_tablet_manager.cc:1397] T f8c76e8bc14e4a31a53e069b5d5b0b93 P c40dc0d1319b43d496ca89db9ca683c4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 02:18:15.803215 12153 raft_consensus.cc:348] T f8c76e8bc14e4a31a53e069b5d5b0b93 P c40dc0d1319b43d496ca89db9ca683c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 } } peers { permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 } } peers { permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 } }
I20260928 02:18:15.803449 12153 raft_consensus.cc:374] T f8c76e8bc14e4a31a53e069b5d5b0b93 P c40dc0d1319b43d496ca89db9ca683c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:15.803555 12153 raft_consensus.cc:729] T f8c76e8bc14e4a31a53e069b5d5b0b93 P c40dc0d1319b43d496ca89db9ca683c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c40dc0d1319b43d496ca89db9ca683c4, State: Initialized, Role: FOLLOWER
I20260928 02:18:15.803835 12153 consensus_queue.cc:260] T f8c76e8bc14e4a31a53e069b5d5b0b93 P c40dc0d1319b43d496ca89db9ca683c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 } } peers { permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 } } peers { permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 } }
I20260928 02:18:15.804366 12155 tablet_bootstrap.cc:492] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1: Bootstrap starting.
I20260928 02:18:15.805505 12157 tablet_bootstrap.cc:492] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47: Bootstrap starting.
I20260928 02:18:15.806777 12155 tablet_bootstrap.cc:654] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:15.806818 12157 tablet_bootstrap.cc:654] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:15.808214 12153 ts_tablet_manager.cc:1428] T f8c76e8bc14e4a31a53e069b5d5b0b93 P c40dc0d1319b43d496ca89db9ca683c4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:18:15.809165 12157 tablet_bootstrap.cc:492] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47: No bootstrap required, opened a new log
I20260928 02:18:15.809326 12157 ts_tablet_manager.cc:1397] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 02:18:15.810328 12157 raft_consensus.cc:348] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 } } peers { permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 } } peers { permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 } }
I20260928 02:18:15.810662 12157 raft_consensus.cc:374] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:15.810761 12157 raft_consensus.cc:729] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1736eee0e104b78ad42edc586263f47, State: Initialized, Role: FOLLOWER
I20260928 02:18:15.811084 12157 consensus_queue.cc:260] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 } } peers { permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 } } peers { permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 } }
I20260928 02:18:15.811650 12059 heartbeater.cc:503] Master 127.4.140.252:45985 was elected leader, sending a full tablet report...
I20260928 02:18:15.811769 12155 tablet_bootstrap.cc:492] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1: No bootstrap required, opened a new log
I20260928 02:18:15.811933 12155 ts_tablet_manager.cc:1397] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260928 02:18:15.813149 12155 raft_consensus.cc:348] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 } } peers { permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 } } peers { permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 } }
I20260928 02:18:15.813417 12155 raft_consensus.cc:374] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:15.813529 12155 raft_consensus.cc:729] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4a4eb5099e4262a21d9704f43f0bf1, State: Initialized, Role: FOLLOWER
I20260928 02:18:15.813777 12155 consensus_queue.cc:260] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 } } peers { permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 } } peers { permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 } }
I20260928 02:18:15.814836 12155 ts_tablet_manager.cc:1428] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 02:18:15.816920 11769 heartbeater.cc:503] Master 127.4.140.252:45985 was elected leader, sending a full tablet report...
I20260928 02:18:15.819242 12157 ts_tablet_manager.cc:1428] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260928 02:18:15.819504 11917 heartbeater.cc:503] Master 127.4.140.252:45985 was elected leader, sending a full tablet report...
I20260928 02:18:15.819636 12160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.808837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.195:37263)
0928 02:18:15.818940 (+ 10103us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.819298 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9920,"threads_started":1}
I20260928 02:18:15.820722 12163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.815608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.193:48465)
0928 02:18:15.820315 (+ 4707us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.820451 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4576,"thread_start_us":4438,"threads_started":1}
I20260928 02:18:15.821810 12163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.821419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.193:45005)
0928 02:18:15.821615 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.821692 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 02:18:15.822660 12164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.816362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.194:35325)
0928 02:18:15.822253 (+ 5891us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.822436 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5747,"thread_start_us":5637,"threads_started":1}
I20260928 02:18:15.822762 12160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.822376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.195:46059)
0928 02:18:15.822541 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.822625 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 02:18:15.823549 12164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.823237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.194:43571)
0928 02:18:15.823379 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.823443 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 02:18:15.885732 12162 raft_consensus.cc:482] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:18:15.885979 12162 raft_consensus.cc:504] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 } } peers { permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 } } peers { permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 } }
I20260928 02:18:15.887521 12162 leader_election.cc:288] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1736eee0e104b78ad42edc586263f47 (127.4.140.194:45109), c40dc0d1319b43d496ca89db9ca683c4 (127.4.140.195:46077)
I20260928 02:18:15.893721 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.887952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:38203 (local address 127.4.140.195:46077)
0928 02:18:15.888136 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:15.888144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:15.888640 (+ 496us) server_negotiation.cc:407] Connection header received
0928 02:18:15.888782 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:15.888789 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:15.888868 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:15.888968 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:15.889495 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.890230 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.892002 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.892791 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.892962 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:15.893077 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:15.893331 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 02:18:15.893476 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:18:15.893821 12165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.887762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:46077 (local address 127.4.140.193:38203)
0928 02:18:15.888503 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.888519 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:15.888655 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:15.888978 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:15.888981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:15.889002 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:15.889352 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:15.889364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.890354 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.890357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:15.891802 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:15.891813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.892978 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.892984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:15.893182 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:15.893205 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:18:15.893284 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:18:15.893379 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":1,"thread_start_us":475,"threads_started":1}
I20260928 02:18:15.893859 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.887801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:51253 (local address 127.4.140.194:45109)
0928 02:18:15.888033 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:15.888040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:15.888189 (+ 149us) server_negotiation.cc:407] Connection header received
0928 02:18:15.888455 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:15.888460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:15.888545 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:15.888638 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:15.889193 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.890017 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.891965 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.892802 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.892978 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:15.893065 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:15.893478 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 02:18:15.893618 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:18:15.894326 12163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.887725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:45109 (local address 127.4.140.193:51253)
0928 02:18:15.888071 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.888092 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:15.888299 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:15.888673 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:15.888678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:15.888723 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:15.889031 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:15.889043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.890148 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.890153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:15.891770 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:15.891782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.893101 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.893129 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:15.893314 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:15.893347 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:18:15.893436 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 02:18:15.893576 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":58}
I20260928 02:18:15.894654 12014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c76e8bc14e4a31a53e069b5d5b0b93" candidate_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c40dc0d1319b43d496ca89db9ca683c4" is_pre_election: true
I20260928 02:18:15.895004 11872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c76e8bc14e4a31a53e069b5d5b0b93" candidate_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1736eee0e104b78ad42edc586263f47" is_pre_election: true
I20260928 02:18:15.895123 12014 raft_consensus.cc:2454] T f8c76e8bc14e4a31a53e069b5d5b0b93 P c40dc0d1319b43d496ca89db9ca683c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d4a4eb5099e4262a21d9704f43f0bf1 in term 0.
I20260928 02:18:15.895251 11872 raft_consensus.cc:2454] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d4a4eb5099e4262a21d9704f43f0bf1 in term 0.
I20260928 02:18:15.895735 11657 leader_election.cc:302] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4a4eb5099e4262a21d9704f43f0bf1, c40dc0d1319b43d496ca89db9ca683c4; no voters:
I20260928 02:18:15.896170 12162 raft_consensus.cc:2781] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:18:15.896297 12162 raft_consensus.cc:482] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:18:15.896399 12162 raft_consensus.cc:3037] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 0 FOLLOWER]: Advancing to term 1
W20260928 02:18:15.898484 11920 tablet.cc:2369] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:18:15.898522 12162 raft_consensus.cc:504] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 } } peers { permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 } } peers { permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 } }
I20260928 02:18:15.899143 12162 leader_election.cc:288] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [CANDIDATE]: Term 1 election: Requested vote from peers f1736eee0e104b78ad42edc586263f47 (127.4.140.194:45109), c40dc0d1319b43d496ca89db9ca683c4 (127.4.140.195:46077)
I20260928 02:18:15.899583 11872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c76e8bc14e4a31a53e069b5d5b0b93" candidate_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1736eee0e104b78ad42edc586263f47"
I20260928 02:18:15.899662 12014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c76e8bc14e4a31a53e069b5d5b0b93" candidate_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c40dc0d1319b43d496ca89db9ca683c4"
I20260928 02:18:15.899775 11872 raft_consensus.cc:3037] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:15.899885 12014 raft_consensus.cc:3037] T f8c76e8bc14e4a31a53e069b5d5b0b93 P c40dc0d1319b43d496ca89db9ca683c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:15.901875 11872 raft_consensus.cc:2454] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d4a4eb5099e4262a21d9704f43f0bf1 in term 1.
I20260928 02:18:15.902139 12014 raft_consensus.cc:2454] T f8c76e8bc14e4a31a53e069b5d5b0b93 P c40dc0d1319b43d496ca89db9ca683c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d4a4eb5099e4262a21d9704f43f0bf1 in term 1.
I20260928 02:18:15.902307 11660 leader_election.cc:302] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4a4eb5099e4262a21d9704f43f0bf1, f1736eee0e104b78ad42edc586263f47; no voters:
I20260928 02:18:15.902630 12162 raft_consensus.cc:2781] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:18:15.903074 12162 raft_consensus.cc:686] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 1 LEADER]: Becoming Leader. State: Replica: 1d4a4eb5099e4262a21d9704f43f0bf1, State: Running, Role: LEADER
I20260928 02:18:15.903759 12162 consensus_queue.cc:237] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 } } peers { permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 } } peers { permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 } }
I20260928 02:18:15.904104 12163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.903619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.193:39937)
0928 02:18:15.903801 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.903932 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 02:18:15.907325 11588 catalog_manager.cc:5495] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d4a4eb5099e4262a21d9704f43f0bf1 (127.4.140.193). New cstate: current_term: 1 leader_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d4a4eb5099e4262a21d9704f43f0bf1" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 35389 } health_report { overall_health: HEALTHY } } }
I20260928 02:18:15.948251 12152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.943205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50910 (local address 127.4.140.193:35389)
0928 02:18:15.943431 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:15.943437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:15.943459 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:18:15.943568 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:15.943574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:15.943660 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:15.943810 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:18:15.944199 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.944900 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.946185 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:15.946593 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:15.946716 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:15.947117 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:18:15.947646 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:18:15.947794 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:15.947886 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 02:18:15.947988 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:18:15.952183 11872 raft_consensus.cc:1261] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47 [term 1 FOLLOWER]: Refusing update from remote peer 1d4a4eb5099e4262a21d9704f43f0bf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:18:15.952437 12014 raft_consensus.cc:1261] T f8c76e8bc14e4a31a53e069b5d5b0b93 P c40dc0d1319b43d496ca89db9ca683c4 [term 1 FOLLOWER]: Refusing update from remote peer 1d4a4eb5099e4262a21d9704f43f0bf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:18:15.952907 12166 consensus_queue.cc:1035] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1736eee0e104b78ad42edc586263f47" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:18:15.953238 12162 consensus_queue.cc:1035] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c40dc0d1319b43d496ca89db9ca683c4" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:18:15.955698 12164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.955115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.194:57269)
0928 02:18:15.955331 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.955478 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 02:18:15.956636 12160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.956125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.195:46681)
0928 02:18:15.956269 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.956391 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 02:18:15.964646 12160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.964119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.195:37309)
0928 02:18:15.964339 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.964430 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260928 02:18:15.965142 12164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.964690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.194:42953)
0928 02:18:15.964913 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.965000 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260928 02:18:15.968603 12163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.967158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.4.140.193:33475)
0928 02:18:15.967512 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:15.967683 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260928 02:18:15.970031 12169 mvcc.cc:205] Tried to move back new op lower bound from 7334141525813239808 to 7334141525625843712. Current Snapshot: MvccSnapshot[applied={T|T < 7334141525813239808 or (T in {7334141525813239808})}]
I20260928 02:18:15.979559 11589 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:18:15.979842 11589 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:18:15.985380 11589 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8106b15bf894b04b0413a7a07fc5198: Sending DeleteTablet for 3 replicas of tablet f8c76e8bc14e4a31a53e069b5d5b0b93
I20260928 02:18:15.986495 11852 tablet_service.cc:1514] Processing DeleteTablet for tablet f8c76e8bc14e4a31a53e069b5d5b0b93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:15 UTC) from {username='oryx', principal='oryx/127.4.140.252@KRBTEST.COM'} at 127.0.0.1:47924
I20260928 02:18:15.986828 11704 tablet_service.cc:1514] Processing DeleteTablet for tablet f8c76e8bc14e4a31a53e069b5d5b0b93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:15 UTC) from {username='oryx', principal='oryx/127.4.140.252@KRBTEST.COM'} at 127.0.0.1:50906
I20260928 02:18:15.986918 11994 tablet_service.cc:1514] Processing DeleteTablet for tablet f8c76e8bc14e4a31a53e069b5d5b0b93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:15 UTC) from {username='oryx', principal='oryx/127.4.140.252@KRBTEST.COM'} at 127.0.0.1:44082
I20260928 02:18:15.987581 12179 tablet_replica.cc:330] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1: stopping tablet replica
I20260928 02:18:15.987579 12180 tablet_replica.cc:330] T f8c76e8bc14e4a31a53e069b5d5b0b93 P c40dc0d1319b43d496ca89db9ca683c4: stopping tablet replica
I20260928 02:18:15.987825 12179 raft_consensus.cc:2229] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:18:15.987823 12180 raft_consensus.cc:2229] T f8c76e8bc14e4a31a53e069b5d5b0b93 P c40dc0d1319b43d496ca89db9ca683c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:18:15.988205 12180 raft_consensus.cc:2258] T f8c76e8bc14e4a31a53e069b5d5b0b93 P c40dc0d1319b43d496ca89db9ca683c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:15.988518 12179 raft_consensus.cc:2258] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:15.989099 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 11590
I20260928 02:18:15.989682 12180 ts_tablet_manager.cc:1905] T f8c76e8bc14e4a31a53e069b5d5b0b93 P c40dc0d1319b43d496ca89db9ca683c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:15.989919 12179 ts_tablet_manager.cc:1905] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:15.997038 12179 ts_tablet_manager.cc:1918] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:18:15.997319 12179 log.cc:1196] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/f8c76e8bc14e4a31a53e069b5d5b0b93
I20260928 02:18:15.997890 12179 ts_tablet_manager.cc:1939] T f8c76e8bc14e4a31a53e069b5d5b0b93 P 1d4a4eb5099e4262a21d9704f43f0bf1: Deleting consensus metadata
I20260928 02:18:15.998036 12180 ts_tablet_manager.cc:1918] T f8c76e8bc14e4a31a53e069b5d5b0b93 P c40dc0d1319b43d496ca89db9ca683c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:18:15.998227 12180 log.cc:1196] T f8c76e8bc14e4a31a53e069b5d5b0b93 P c40dc0d1319b43d496ca89db9ca683c4: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/f8c76e8bc14e4a31a53e069b5d5b0b93
I20260928 02:18:15.998771 12180 ts_tablet_manager.cc:1939] T f8c76e8bc14e4a31a53e069b5d5b0b93 P c40dc0d1319b43d496ca89db9ca683c4: Deleting consensus metadata
I20260928 02:18:15.999424 11576 catalog_manager.cc:4841] TS 1d4a4eb5099e4262a21d9704f43f0bf1 (127.4.140.193:35389): tablet f8c76e8bc14e4a31a53e069b5d5b0b93 (table test-table [id=f5b39dc3b1924763927b386baaaee4f3]) successfully deleted
I20260928 02:18:15.999980 11573 catalog_manager.cc:4841] TS c40dc0d1319b43d496ca89db9ca683c4 (127.4.140.195:46077): tablet f8c76e8bc14e4a31a53e069b5d5b0b93 (table test-table [id=f5b39dc3b1924763927b386baaaee4f3]) successfully deleted
I20260928 02:18:16.000398 12178 tablet_replica.cc:330] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47: stopping tablet replica
I20260928 02:18:16.000610 12178 raft_consensus.cc:2229] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:18:16.000908 12178 raft_consensus.cc:2258] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:16.001978 12178 ts_tablet_manager.cc:1905] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:16.008590 12178 ts_tablet_manager.cc:1918] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:18:16.008733 12178 log.cc:1196] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/f8c76e8bc14e4a31a53e069b5d5b0b93
I20260928 02:18:16.009167 12178 ts_tablet_manager.cc:1939] T f8c76e8bc14e4a31a53e069b5d5b0b93 P f1736eee0e104b78ad42edc586263f47: Deleting consensus metadata
I20260928 02:18:16.010310 11576 catalog_manager.cc:4841] TS f1736eee0e104b78ad42edc586263f47 (127.4.140.194:45109): tablet f8c76e8bc14e4a31a53e069b5d5b0b93 (table test-table [id=f5b39dc3b1924763927b386baaaee4f3]) successfully deleted
I20260928 02:18:16.026001 12152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:15.990687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.4.140.193:35389)
0928 02:18:15.990852 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:15.990858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:15.993600 (+ 2742us) server_negotiation.cc:407] Connection header received
0928 02:18:16.004069 (+ 10469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:16.004074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:16.004171 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:16.004295 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:16.005644 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:16.006483 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:16.007750 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:16.008288 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:16.008444 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:16.020143 (+ 11699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:16.020164 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:16.020185 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:16.020254 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:16.023275 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:16.023971 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:16.023979 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:16.023981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:16.024134 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:16.024507 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:16.024515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:16.024518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:16.024852 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:16.025175 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:16.025424 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 02:18:16.025661 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:18:16.026677 11764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:16.129807 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 11590
I20260928 02:18:16.151674 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 11781
I20260928 02:18:16.167098 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:16.152595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.4.140.194:45109)
0928 02:18:16.152763 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:16.152769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:16.152790 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:18:16.152925 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:16.152930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:16.153001 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:16.153098 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:16.154495 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:16.155695 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:16.157133 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:16.157804 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:16.157921 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:16.160582 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:16.160602 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:16.160622 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:16.160676 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:16.164580 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:16.165184 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:16.165195 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:16.165199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:16.165337 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:16.165671 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:16.165677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:16.165682 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:16.166105 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:16.166399 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:16.166553 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 02:18:16.166751 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:18:16.167804 11912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:16.247454 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:16.246905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.0.0.1:37230)
0928 02:18:16.247127 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:16.247245 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 02:18:16.248605 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:16.248206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36419 (local address 127.0.0.1:37244)
0928 02:18:16.248364 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:16.248450 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 02:18:16.260233 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 11781
I20260928 02:18:16.278532 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 11930
I20260928 02:18:16.291319 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:16.279293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44096 (local address 127.4.140.195:46077)
0928 02:18:16.279474 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:16.279479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:16.279498 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:18:16.279626 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:16.279630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:16.279698 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:16.279779 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:16.281092 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:16.281802 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:16.282952 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:16.283524 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:16.283622 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:16.286069 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:16.286083 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:16.286099 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:16.286130 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:16.289027 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:16.289631 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:16.289638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:16.289640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:16.289762 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:16.290049 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:16.290052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:16.290054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:16.290383 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:16.290604 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:16.290789 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 02:18:16.290939 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:18:16.292052 12054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:16.409245 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 11930
I20260928 02:18:16.432394 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 11259
I20260928 02:18:16.433488 11432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:16.549192 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 11259
I20260928 02:18:16.569468 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 11358
I20260928 02:18:16.570430 11620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:16.688982 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 11358
2026-09-28T02:18:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9535 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:18:16 dist-test-slave-h37r krb5kdc[12189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:18:16 dist-test-slave-h37r krb5kdc[12189](info): set up 2 sockets
Sep 28 02:18:16 dist-test-slave-h37r krb5kdc[12189](info): commencing operation
krb5kdc: starting...
W20260928 02:18:17.784878 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:18:17 dist-test-slave-h37r krb5kdc[12189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:18:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:18:17Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.254.keytab.
Entry for principal oryx/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.254.keytab.
I20260928 02:18:18.009656 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:44903
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.254.keytab
--principal=oryx/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:44903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:18.334582 12202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:18.334931 12202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:18.335089 12202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:18.342489 12202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:18:18.342651 12202 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:18:18.342746 12202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:18.342864 12202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:18.342947 12202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:18:18.343034 12202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:18:18.353129 12202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:44903
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.254.keytab
--principal=oryx/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:44903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:18.354491 12202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:18.355932 12202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:18.368086 12207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:18.368093 12208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:18.368201 12210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:18.369202 12202 server_base.cc:1034] running on GCE node
I20260928 02:18:18.369927 12202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:18.371320 12202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:18.372537 12202 hybrid_clock.cc:648] HybridClock initialized: now 1790561898372460 us; error 42 us; skew 500 ppm
I20260928 02:18:18.372829 12202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:18 dist-test-slave-h37r krb5kdc[12189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561898, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:18.376000 12202 init.cc:377] Logged in from keytab as oryx/127.4.140.254@KRBTEST.COM (short username oryx)
I20260928 02:18:18.377468 12202 webserver.cc:460] Webserver started at http://127.4.140.254:38973/ using document root <none> and password file <none>
I20260928 02:18:18.378178 12202 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:18.378288 12202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:18.378628 12202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:18.381619 12202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "72974f12c2e140c1aeb70bd3d355c05c"
format_stamp: "Formatted at 2026-09-28 02:18:18 on dist-test-slave-h37r"
server_key: "804f619036cece4ead7fcaec4cd6ce81"
server_key_iv: "cba4f6667ee2d21af1b9438bbc2cf4df"
server_key_version: "encryptionkey@0"
I20260928 02:18:18.382385 12202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "72974f12c2e140c1aeb70bd3d355c05c"
format_stamp: "Formatted at 2026-09-28 02:18:18 on dist-test-slave-h37r"
server_key: "804f619036cece4ead7fcaec4cd6ce81"
server_key_iv: "cba4f6667ee2d21af1b9438bbc2cf4df"
server_key_version: "encryptionkey@0"
I20260928 02:18:18.387478 12202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:18:18.391044 12217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:18.392485 12202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:18:18.392695 12202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "72974f12c2e140c1aeb70bd3d355c05c"
format_stamp: "Formatted at 2026-09-28 02:18:18 on dist-test-slave-h37r"
server_key: "804f619036cece4ead7fcaec4cd6ce81"
server_key_iv: "cba4f6667ee2d21af1b9438bbc2cf4df"
server_key_version: "encryptionkey@0"
I20260928 02:18:18.392915 12202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:18.418853 12202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:18.422894 12202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:18.423169 12202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:18.444648 12202 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:44903
I20260928 02:18:18.444669 12278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:44903 every 8 connection(s)
I20260928 02:18:18.446050 12202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:18:18.449784 12279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:18.450649 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 12202
I20260928 02:18:18.450874 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:18:18.451486 4659 external_mini_cluster.cc:1428] Setting key aa654bba1ce4e4648755e0c666fce4ab
I20260928 02:18:18.464305 12279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c: Bootstrap starting.
Sep 28 02:18:18 dist-test-slave-h37r krb5kdc[12189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM
I20260928 02:18:18.468292 12279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:18.469626 12279 log.cc:824] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:18.473008 12279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c: No bootstrap required, opened a new log
I20260928 02:18:18.475937 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:18.453559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.4.140.254:44903)
0928 02:18:18.454380 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:18.454392 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:18.454437 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:18:18.456086 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:18.456113 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:18.456532 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:18.456861 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:18.458892 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:18.460307 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:18.462228 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:18.462890 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:18.463042 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:18.468084 (+ 5042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:18.468093 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:18.468117 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:18.468164 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:18.471494 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:18.472282 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:18.472289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:18.472295 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:18.472538 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:18.472914 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:18.472920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:18.472923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:18.473661 (+ 738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:18.473953 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:18.474411 (+ 458us) server_negotiation.cc:299] Negotiation successful
0928 02:18:18.474849 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":401,"threads_started":1}
I20260928 02:18:18.481339 12279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72974f12c2e140c1aeb70bd3d355c05c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 44903 } }
I20260928 02:18:18.481740 12279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:18.481869 12279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72974f12c2e140c1aeb70bd3d355c05c, State: Initialized, Role: FOLLOWER
I20260928 02:18:18.482599 12279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72974f12c2e140c1aeb70bd3d355c05c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 44903 } }
I20260928 02:18:18.482934 12279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:18:18.483093 12279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:18:18.483282 12279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:18.485752 12279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72974f12c2e140c1aeb70bd3d355c05c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 44903 } }
I20260928 02:18:18.486241 12279 leader_election.cc:302] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72974f12c2e140c1aeb70bd3d355c05c; no voters:
I20260928 02:18:18.486786 12279 leader_election.cc:288] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:18:18.486928 12284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:18:18.487684 12284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c [term 1 LEADER]: Becoming Leader. State: Replica: 72974f12c2e140c1aeb70bd3d355c05c, State: Running, Role: LEADER
I20260928 02:18:18.488258 12284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72974f12c2e140c1aeb70bd3d355c05c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 44903 } }
I20260928 02:18:18.489058 12279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:18:18.491159 12285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72974f12c2e140c1aeb70bd3d355c05c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72974f12c2e140c1aeb70bd3d355c05c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 44903 } } }
I20260928 02:18:18.491219 12286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72974f12c2e140c1aeb70bd3d355c05c. Latest consensus state: current_term: 1 leader_uuid: "72974f12c2e140c1aeb70bd3d355c05c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72974f12c2e140c1aeb70bd3d355c05c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 44903 } } }
I20260928 02:18:18.491526 12285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:18.491554 12286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:18.494356 12289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:18:18.499598 12289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:18:18.510077 12289 catalog_manager.cc:1347] Generated new cluster ID: 99dae3b531994409bb9d6b1c7cc562db
I20260928 02:18:18.510206 12289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:18:18.534958 12289 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:18:18.536397 12289 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:18:18.544428 12289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c: Generated new TSK 0
I20260928 02:18:18.545367 12289 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.193.keytab.
Entry for principal oryx/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.193.keytab.
I20260928 02:18:18.607156 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:44903
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.193.keytab
--principal=oryx/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:18.980644 12305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:18.981012 12305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:18.981137 12305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:18.988546 12305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:18.988660 12305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:18.988745 12305 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:18:18.988826 12305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:18:18.999368 12305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.193.keytab
--principal=oryx/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.140.254:44903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:19.000717 12305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:19.002090 12305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:19.015600 12310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:19.015698 12313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:19.015846 12311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:19.016971 12305 server_base.cc:1034] running on GCE node
I20260928 02:18:19.017675 12305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:19.018728 12305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:19.020023 12305 hybrid_clock.cc:648] HybridClock initialized: now 1790561899019935 us; error 52 us; skew 500 ppm
I20260928 02:18:19.020351 12305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:19 dist-test-slave-h37r krb5kdc[12189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561899, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:19.023847 12305 init.cc:377] Logged in from keytab as oryx/127.4.140.193@KRBTEST.COM (short username oryx)
I20260928 02:18:19.025360 12305 webserver.cc:460] Webserver started at http://127.4.140.193:41413/ using document root <none> and password file <none>
I20260928 02:18:19.026103 12305 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:19.026240 12305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:19.026583 12305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:19.029760 12305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "28fa0e82e3e0496c89d30fdb80df848f"
format_stamp: "Formatted at 2026-09-28 02:18:19 on dist-test-slave-h37r"
server_key: "4c7e76d4dbf18789e813f2ff3d53234d"
server_key_iv: "49a15c7db7e20fb608394cfe69e1ab31"
server_key_version: "encryptionkey@0"
I20260928 02:18:19.030622 12305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "28fa0e82e3e0496c89d30fdb80df848f"
format_stamp: "Formatted at 2026-09-28 02:18:19 on dist-test-slave-h37r"
server_key: "4c7e76d4dbf18789e813f2ff3d53234d"
server_key_iv: "49a15c7db7e20fb608394cfe69e1ab31"
server_key_version: "encryptionkey@0"
I20260928 02:18:19.036190 12305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 02:18:19.039813 12320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:19.041520 12305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:18:19.041709 12305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "28fa0e82e3e0496c89d30fdb80df848f"
format_stamp: "Formatted at 2026-09-28 02:18:19 on dist-test-slave-h37r"
server_key: "4c7e76d4dbf18789e813f2ff3d53234d"
server_key_iv: "49a15c7db7e20fb608394cfe69e1ab31"
server_key_version: "encryptionkey@0"
I20260928 02:18:19.041947 12305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:19.055088 12305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:19.059581 12305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:19.059823 12305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:19.062589 12305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:19.062719 12305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:19.062846 12305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:19.062932 12305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:18:19 dist-test-slave-h37r krb5kdc[12189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561899, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM
I20260928 02:18:19.089110 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:19.063324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:53981 (local address 127.4.140.254:44903)
0928 02:18:19.063856 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:19.063861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:19.065107 (+ 1246us) server_negotiation.cc:407] Connection header received
0928 02:18:19.067139 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:19.067144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:19.067236 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:19.067350 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:19.069457 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:19.070797 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:19.073348 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:19.073885 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:19.074052 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:19.079786 (+ 5734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:19.079792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:19.079796 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:19.079834 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:19.084397 (+ 4563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:19.085615 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:19.085620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:19.085624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:19.085825 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:19.086622 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:19.086625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:19.086628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:19.086887 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:19.087004 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:19.088340 (+ 1336us) server_negotiation.cc:299] Negotiation successful
0928 02:18:19.088626 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":322,"threads_started":1}
I20260928 02:18:19.091315 12330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:19.064051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:44903 (local address 127.4.140.193:53981)
0928 02:18:19.064945 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:19.064997 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:19.066854 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:19.067586 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:19.067598 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:19.068356 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:19.069217 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:19.069255 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:19.070975 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:19.070987 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:19.073138 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:19.073155 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:19.074225 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:19.074233 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:19.074555 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:19.075547 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:19.075580 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:19.079489 (+ 3909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:19.084626 (+ 5137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:19.084634 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:19.084648 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:19.085462 (+ 814us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:19.085982 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:19.085994 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:19.086000 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:19.086407 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:19.087044 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:19.087058 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:19.087735 (+ 677us) client_negotiation.cc:769] Sending connection context
0928 02:18:19.088204 (+ 469us) client_negotiation.cc:240] Negotiation successful
0928 02:18:19.089107 (+ 903us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":423,"threads_started":1}
I20260928 02:18:19.113282 12305 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:41081
I20260928 02:18:19.113296 12440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:41081 every 8 connection(s)
I20260928 02:18:19.114974 12305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:18:19.115860 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 12305
I20260928 02:18:19.116156 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:18:19.116961 4659 external_mini_cluster.cc:1428] Setting key 66545cfef1dbada3c239d8d517790967
I20260928 02:18:19.117724 12441 heartbeater.cc:347] Connected to a master server at 127.4.140.254:44903
I20260928 02:18:19.118368 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:19.098821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:58279 (local address 127.4.140.254:44903)
0928 02:18:19.099018 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:19.099024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:19.100515 (+ 1491us) server_negotiation.cc:407] Connection header received
0928 02:18:19.100871 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:19.100880 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:19.100976 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:19.101060 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:19.103010 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:19.104330 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:19.106735 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:19.107557 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:19.107699 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:19.110985 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:19.110994 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:19.111002 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:19.111053 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:19.114503 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:19.115223 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:19.115230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:19.115234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:19.115417 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:19.115962 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:19.115981 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:19.116030 (+ 49us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:19.116467 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:19.116631 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:19.117941 (+ 1310us) server_negotiation.cc:299] Negotiation successful
0928 02:18:19.118123 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:18:19.119061 12400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:19.099107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:44903 (local address 127.4.140.193:58279)
0928 02:18:19.100380 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:19.100406 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:19.100671 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:19.101343 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:19.101351 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:19.102180 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:19.102793 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:19.102814 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:19.104518 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:19.104523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:19.106515 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:19.106537 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:19.108094 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:19.108105 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:19.108363 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:19.109193 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:19.109234 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:19.110702 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:19.114646 (+ 3944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:19.114650 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:19.114653 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:19.115087 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:19.115590 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:19.115598 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:19.115605 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:19.115796 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:19.116698 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:19.116704 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:19.116922 (+ 218us) client_negotiation.cc:769] Sending connection context
0928 02:18:19.117122 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 02:18:19.117452 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"thread_start_us":855,"threads_started":1}
I20260928 02:18:19.119468 12441 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:19.120572 12441 heartbeater.cc:511] Master 127.4.140.254:44903 requested a full tablet report, sending...
I20260928 02:18:19.123389 12234 ts_manager.cc:194] Registered new tserver with Master: 28fa0e82e3e0496c89d30fdb80df848f (127.4.140.193:41081)
I20260928 02:18:19.126062 12234 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:53981
WARNING: no policy specified for oryx/127.4.140.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.140.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.194.keytab.
Entry for principal oryx/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.194.keytab.
I20260928 02:18:19.167083 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:44903
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.194.keytab
--principal=oryx/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:19.497840 12445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:19.498142 12445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:19.498298 12445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:19.505601 12445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:19.505717 12445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:19.505853 12445 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:18:19.505978 12445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:18:19.516000 12445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.194.keytab
--principal=oryx/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.140.254:44903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:19.517336 12445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:19.518644 12445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:19.531507 12450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:19.531585 12453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:19.531541 12451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:19.532750 12445 server_base.cc:1034] running on GCE node
I20260928 02:18:19.533362 12445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:19.534379 12445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:19.535616 12445 hybrid_clock.cc:648] HybridClock initialized: now 1790561899535534 us; error 56 us; skew 500 ppm
I20260928 02:18:19.535969 12445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:19 dist-test-slave-h37r krb5kdc[12189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561899, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:19.539018 12445 init.cc:377] Logged in from keytab as oryx/127.4.140.194@KRBTEST.COM (short username oryx)
I20260928 02:18:19.540602 12445 webserver.cc:460] Webserver started at http://127.4.140.194:42719/ using document root <none> and password file <none>
I20260928 02:18:19.541340 12445 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:19.541465 12445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:19.541808 12445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:19.544979 12445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "20f2f1dffc9a478cb5182a25c66a1a9a"
format_stamp: "Formatted at 2026-09-28 02:18:19 on dist-test-slave-h37r"
server_key: "75fba073c4a498bd00d97b48738ef30c"
server_key_iv: "011131d0ad1003e97cf7fa6332753737"
server_key_version: "encryptionkey@0"
I20260928 02:18:19.545807 12445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "20f2f1dffc9a478cb5182a25c66a1a9a"
format_stamp: "Formatted at 2026-09-28 02:18:19 on dist-test-slave-h37r"
server_key: "75fba073c4a498bd00d97b48738ef30c"
server_key_iv: "011131d0ad1003e97cf7fa6332753737"
server_key_version: "encryptionkey@0"
I20260928 02:18:19.550959 12445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:18:19.554410 12460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:19.555768 12445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:18:19.555976 12445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "20f2f1dffc9a478cb5182a25c66a1a9a"
format_stamp: "Formatted at 2026-09-28 02:18:19 on dist-test-slave-h37r"
server_key: "75fba073c4a498bd00d97b48738ef30c"
server_key_iv: "011131d0ad1003e97cf7fa6332753737"
server_key_version: "encryptionkey@0"
I20260928 02:18:19.556205 12445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:19.574175 12445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:19.578073 12445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:19.578336 12445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:19.580978 12445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:19.581099 12445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:19.581243 12445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:19.581305 12445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:18:19 dist-test-slave-h37r krb5kdc[12189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561899, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM
I20260928 02:18:19.603345 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:19.581309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:45199 (local address 127.4.140.254:44903)
0928 02:18:19.581508 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:19.581514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:19.582907 (+ 1393us) server_negotiation.cc:407] Connection header received
0928 02:18:19.584923 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:19.584929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:19.585018 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:19.585125 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:19.587343 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:19.588469 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:19.590219 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:19.590709 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:19.590839 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:19.596127 (+ 5288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:19.596135 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:19.596143 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:19.596205 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:19.599591 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:19.600658 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:19.600667 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:19.600673 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:19.600822 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:19.601273 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:19.601278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:19.601281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:19.601632 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:19.601814 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:19.602908 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0928 02:18:19.603104 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:18:19.605800 12469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:19.581868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:44903 (local address 127.4.140.194:45199)
0928 02:18:19.582735 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:19.582785 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:19.584648 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:19.585354 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:19.585362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:19.586122 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:19.587116 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:19.587149 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:19.588648 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:19.588652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:19.590074 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:19.590090 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:19.590939 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:19.590947 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:19.591170 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:19.591912 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:19.591948 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:19.595828 (+ 3880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:19.599740 (+ 3912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:19.599749 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:19.599767 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:19.600454 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:19.600923 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:19.600928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:19.600932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:19.601155 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:19.601791 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:19.601805 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:19.602416 (+ 611us) client_negotiation.cc:769] Sending connection context
0928 02:18:19.602812 (+ 396us) client_negotiation.cc:240] Negotiation successful
0928 02:18:19.603562 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":374,"threads_started":1}
I20260928 02:18:19.626431 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:19.611411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:35963 (local address 127.4.140.254:44903)
0928 02:18:19.611976 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:19.611983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:19.612531 (+ 548us) server_negotiation.cc:407] Connection header received
0928 02:18:19.612754 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:19.612757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:19.612818 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:19.612891 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:19.614349 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:19.615031 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:19.617447 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:19.617907 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:19.618056 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:19.619985 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:19.619989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:19.619993 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:19.620028 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:19.623178 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:19.624267 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:19.624271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:19.624273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:19.624392 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:19.624790 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:19.624797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:19.624802 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:19.625261 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:19.625444 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:19.626030 (+ 586us) server_negotiation.cc:299] Negotiation successful
0928 02:18:19.626202 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396}
I20260928 02:18:19.626991 12529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:19.611525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:44903 (local address 127.4.140.194:35963)
0928 02:18:19.612398 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:19.612421 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:19.612617 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:19.612955 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:19.612963 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:19.613717 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:19.614197 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:19.614213 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:19.615175 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:19.615183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:19.617267 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:19.617283 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:19.618103 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:19.618109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:19.618285 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:19.618959 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:19.618975 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:19.619792 (+ 817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:19.623338 (+ 3546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:19.623343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:19.623355 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:19.624119 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:19.624504 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:19.624510 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:19.624513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:19.624655 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:19.625434 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:19.625442 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:19.625710 (+ 268us) client_negotiation.cc:769] Sending connection context
0928 02:18:19.625903 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 02:18:19.626343 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":490,"threads_started":1}
I20260928 02:18:19.640064 12445 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:45669
I20260928 02:18:19.640112 12579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:45669 every 8 connection(s)
I20260928 02:18:19.641299 12445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:18:19.643370 12580 heartbeater.cc:347] Connected to a master server at 127.4.140.254:44903
I20260928 02:18:19.643661 12580 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:19.644349 12580 heartbeater.cc:511] Master 127.4.140.254:44903 requested a full tablet report, sending...
I20260928 02:18:19.645843 12234 ts_manager.cc:194] Registered new tserver with Master: 20f2f1dffc9a478cb5182a25c66a1a9a (127.4.140.194:45669)
I20260928 02:18:19.646747 12234 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:45199
I20260928 02:18:19.648515 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 12445
I20260928 02:18:19.648713 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:18:19.649164 4659 external_mini_cluster.cc:1428] Setting key 5fd18a59ee8eb2972af3516259a4d926
WARNING: no policy specified for oryx/127.4.140.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.140.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.195.keytab.
Entry for principal oryx/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.195.keytab.
I20260928 02:18:19.695835 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:44903
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.195.keytab
--principal=oryx/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:20.030377 12584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:20.030654 12584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:20.030750 12584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:20.037873 12584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:20.037969 12584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:20.038051 12584 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:18:20.038130 12584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:18:20.047667 12584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.195.keytab
--principal=oryx/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.140.254:44903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:20.048899 12584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:20.050140 12584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:20.063212 12589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:20.063277 12590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:20.063316 12592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:20.063946 12584 server_base.cc:1034] running on GCE node
I20260928 02:18:20.064525 12584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:20.065346 12584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:20.066598 12584 hybrid_clock.cc:648] HybridClock initialized: now 1790561900066517 us; error 56 us; skew 500 ppm
I20260928 02:18:20.066912 12584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:20 dist-test-slave-h37r krb5kdc[12189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561900, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:20.070016 12584 init.cc:377] Logged in from keytab as oryx/127.4.140.195@KRBTEST.COM (short username oryx)
I20260928 02:18:20.071472 12584 webserver.cc:460] Webserver started at http://127.4.140.195:39039/ using document root <none> and password file <none>
I20260928 02:18:20.072213 12584 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:20.072333 12584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:20.072639 12584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:20.075429 12584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "86f236333f4148b886a2e7a81af6dff8"
format_stamp: "Formatted at 2026-09-28 02:18:20 on dist-test-slave-h37r"
server_key: "2f5587608f9391227100149c340f8e6e"
server_key_iv: "28c801021e50b218ddd3d375d7d52807"
server_key_version: "encryptionkey@0"
I20260928 02:18:20.076289 12584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "86f236333f4148b886a2e7a81af6dff8"
format_stamp: "Formatted at 2026-09-28 02:18:20 on dist-test-slave-h37r"
server_key: "2f5587608f9391227100149c340f8e6e"
server_key_iv: "28c801021e50b218ddd3d375d7d52807"
server_key_version: "encryptionkey@0"
I20260928 02:18:20.081246 12584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:18:20.084734 12599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:20.086050 12584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:18:20.086217 12584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "86f236333f4148b886a2e7a81af6dff8"
format_stamp: "Formatted at 2026-09-28 02:18:20 on dist-test-slave-h37r"
server_key: "2f5587608f9391227100149c340f8e6e"
server_key_iv: "28c801021e50b218ddd3d375d7d52807"
server_key_version: "encryptionkey@0"
I20260928 02:18:20.086428 12584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:20.103027 12584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:20.107353 12584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:20.107656 12584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:20.110553 12584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:20.110651 12584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:20.110782 12584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:20.110878 12584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:18:20 dist-test-slave-h37r krb5kdc[12189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561900, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM
I20260928 02:18:20.129493 12441 heartbeater.cc:503] Master 127.4.140.254:44903 was elected leader, sending a full tablet report...
I20260928 02:18:20.135784 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:20.112107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:52987 (local address 127.4.140.254:44903)
0928 02:18:20.112337 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:20.112346 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:20.112664 (+ 318us) server_negotiation.cc:407] Connection header received
0928 02:18:20.114725 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:20.114731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:20.114819 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:20.114930 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:20.117471 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:20.118730 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:20.121710 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:20.122104 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:20.122243 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:20.127526 (+ 5283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:20.127530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:20.127534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:20.127566 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:20.132139 (+ 4573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:20.133024 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:20.133033 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:20.133039 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:20.133230 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:20.133635 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:20.133643 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:20.133649 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:20.134092 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:20.134271 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:20.134996 (+ 725us) server_negotiation.cc:299] Negotiation successful
0928 02:18:20.135288 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:18:20.136896 12608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:20.111631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:44903 (local address 127.4.140.195:52987)
0928 02:18:20.112494 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:20.112539 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:20.114433 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:20.115324 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:20.115331 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:20.116106 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:20.117217 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:20.117252 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:20.118950 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:20.118957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:20.121535 (+ 2578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:20.121557 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:20.122346 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:20.122353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:20.122686 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:20.123666 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:20.123711 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:20.127296 (+ 3585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:20.132312 (+ 5016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:20.132322 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:20.132339 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:20.132889 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:20.133347 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:20.133350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:20.133352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:20.133521 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:20.134213 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:20.134219 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:20.134552 (+ 333us) client_negotiation.cc:769] Sending connection context
0928 02:18:20.134806 (+ 254us) client_negotiation.cc:240] Negotiation successful
0928 02:18:20.135440 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":466,"threads_started":1}
I20260928 02:18:20.158362 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:20.143947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:44903 (local address 127.4.140.195:37189)
0928 02:18:20.144755 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:20.144771 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:20.144878 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:20.145254 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:20.145257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:20.145702 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:20.146018 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:20.146027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:20.147751 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:20.147756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:20.149354 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:20.149366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:20.150076 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:20.150082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:20.150253 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:20.150863 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:20.150879 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:20.151564 (+ 685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:20.155071 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:20.155079 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:20.155084 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:20.155524 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:20.156007 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:20.156015 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:20.156020 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:20.156207 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:20.156904 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:20.156907 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:20.157185 (+ 278us) client_negotiation.cc:769] Sending connection context
0928 02:18:20.157369 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 02:18:20.157771 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":498,"threads_started":1}
I20260928 02:18:20.159055 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:20.143933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:37189 (local address 127.4.140.254:44903)
0928 02:18:20.144132 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:20.144138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:20.144879 (+ 741us) server_negotiation.cc:407] Connection header received
0928 02:18:20.144976 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:20.144984 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:20.145084 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:20.145212 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:20.146156 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:20.147594 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:20.149495 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:20.149898 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:20.150036 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:20.151743 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:20.151748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:20.151751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:20.151779 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:20.154939 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:20.155674 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:20.155683 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:20.155689 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:20.155850 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:20.156327 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:20.156332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:20.156336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:20.156699 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:20.156846 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:20.158437 (+ 1591us) server_negotiation.cc:299] Negotiation successful
0928 02:18:20.158705 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:18:20.169926 12584 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:32857
I20260928 02:18:20.169934 12718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:32857 every 8 connection(s)
I20260928 02:18:20.171108 12584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:18:20.173126 12719 heartbeater.cc:347] Connected to a master server at 127.4.140.254:44903
I20260928 02:18:20.173349 12719 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:20.173964 12719 heartbeater.cc:511] Master 127.4.140.254:44903 requested a full tablet report, sending...
I20260928 02:18:20.175210 12234 ts_manager.cc:194] Registered new tserver with Master: 86f236333f4148b886a2e7a81af6dff8 (127.4.140.195:32857)
I20260928 02:18:20.176100 12234 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:52987
I20260928 02:18:20.178256 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 12584
I20260928 02:18:20.178469 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:18:20.178895 4659 external_mini_cluster.cc:1428] Setting key 057fad4aa5b9bb085b2a3eb61e25a444
I20260928 02:18:20.181658 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:18:20 dist-test-slave-h37r krb5kdc[12189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561900, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:18:20 dist-test-slave-h37r krb5kdc[12189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:20 dist-test-slave-h37r krb5kdc[12189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:20.649187 12580 heartbeater.cc:503] Master 127.4.140.254:44903 was elected leader, sending a full tablet report...
I20260928 02:18:21.178601 12719 heartbeater.cc:503] Master 127.4.140.254:44903 was elected leader, sending a full tablet report...
W20260928 02:18:21.316309 12727 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.089s user 0.004s sys 0.011s
W20260928 02:18:21.317389 12329 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:20.221385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43544 (local address 127.4.140.254:44903)
0928 02:18:20.221555 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:20.221559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:20.221790 (+ 231us) server_negotiation.cc:407] Connection header received
0928 02:18:20.222132 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:20.222137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:20.222214 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:20.222330 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:20.223721 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:20.224521 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:20.225786 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:20.226204 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:20.226297 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:21.317163 (+1090866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43544: BlockingRecv error: recv got EOF from 127.0.0.1:43544 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 02:18:21 dist-test-slave-h37r krb5kdc[12189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561900, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM
I20260928 02:18:21.335580 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:21.321580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43546 (local address 127.4.140.254:44903)
0928 02:18:21.321793 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:21.321800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:21.321963 (+ 163us) server_negotiation.cc:407] Connection header received
0928 02:18:21.322195 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:21.322201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:21.322303 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:21.322410 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:21.323626 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.324400 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.326101 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.326503 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.326636 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:21.330540 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:21.330546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:21.330549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:21.330584 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:21.333256 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:21.333874 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:21.333878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:21.333880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:21.334009 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:21.334319 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:21.334322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:21.334324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:21.334615 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:21.334921 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:21.335017 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 02:18:21.335207 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:18:21.340085 12234 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:18:21.348302 12234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:18:21.379438 12738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:21.360430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:41081 (local address 127.0.0.1:43912)
0928 02:18:21.361655 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:21.361703 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:21.361927 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:21.362973 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:21.362981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:21.363012 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:21.364960 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:21.364977 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.369302 (+ 4325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.369311 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:21.373914 (+ 4603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:21.373935 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.376963 (+ 3028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.376973 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:21.378484 (+ 1511us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:21.378559 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 02:18:21.378731 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 02:18:21.378974 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"mutex_wait_us":28,"thread_start_us":795,"threads_started":1}
I20260928 02:18:21.380324 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:21.360674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43912 (local address 127.4.140.193:41081)
0928 02:18:21.361384 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:21.361392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:21.362140 (+ 748us) server_negotiation.cc:407] Connection header received
0928 02:18:21.362243 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:21.362251 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:21.362511 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:21.362730 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:21.365230 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.369106 (+ 3876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.374169 (+ 5063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.376329 (+ 2160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.376516 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:21.376804 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:21.379451 (+ 2647us) server_negotiation.cc:299] Negotiation successful
0928 02:18:21.380038 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":364,"threads_started":1}
I20260928 02:18:21.383847 12371 tablet_service.cc:1467] Processing CreateTablet for tablet 4dadea5cb4f040d99da1afd148f8fabf (DEFAULT_TABLE table=test-table [id=d367eca5dd6a48f3b0099e9894c4d724]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:21.385452 12371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dadea5cb4f040d99da1afd148f8fabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:21.385673 12741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:21.362066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.4.140.195:32857)
0928 02:18:21.377386 (+ 15320us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:21.377400 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:21.377442 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:18:21.377587 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:21.377599 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:21.377976 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:21.378225 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:21.379040 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.380849 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.383230 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.384638 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.384786 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:21.384980 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:21.385218 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 02:18:21.385373 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15073,"thread_start_us":17086,"threads_started":1}
I20260928 02:18:21.385674 12740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:21.361536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:32857 (local address 127.0.0.1:43898)
0928 02:18:21.362118 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:21.362136 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:21.362259 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:21.378203 (+ 15944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:21.378208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:21.378234 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:21.378829 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:21.378841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.380983 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.380988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:21.383047 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:21.383062 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.384844 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.384850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:21.385009 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:21.385052 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:18:21.385154 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:18:21.385288 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":77,"thread_start_us":265,"threads_started":1}
I20260928 02:18:21.387244 12742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:21.362639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:45669 (local address 127.0.0.1:55814)
0928 02:18:21.378888 (+ 16249us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:21.378904 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:21.379059 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:21.379975 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:21.379980 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:21.380005 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:21.380312 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:21.380320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.382097 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.382103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:21.384394 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:21.384405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.386457 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.386462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:21.386622 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:21.386654 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:18:21.386745 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 02:18:21.386885 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16114,"mutex_wait_us":1,"thread_start_us":15959,"threads_started":1}
I20260928 02:18:21.387349 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:21.362884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55814 (local address 127.4.140.194:45669)
0928 02:18:21.378073 (+ 15189us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:21.378083 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:21.379015 (+ 932us) server_negotiation.cc:407] Connection header received
0928 02:18:21.379425 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:21.379434 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:21.379735 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:21.379983 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:21.380515 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.381900 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.384599 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.386260 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.386429 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:21.386638 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:21.386913 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 02:18:21.387079 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15022,"thread_start_us":15200,"threads_started":1}
I20260928 02:18:21.389605 12649 tablet_service.cc:1467] Processing CreateTablet for tablet 4dadea5cb4f040d99da1afd148f8fabf (DEFAULT_TABLE table=test-table [id=d367eca5dd6a48f3b0099e9894c4d724]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:21.390731 12510 tablet_service.cc:1467] Processing CreateTablet for tablet 4dadea5cb4f040d99da1afd148f8fabf (DEFAULT_TABLE table=test-table [id=d367eca5dd6a48f3b0099e9894c4d724]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:21.391871 12649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dadea5cb4f040d99da1afd148f8fabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:21.392071 12510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dadea5cb4f040d99da1afd148f8fabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:21.398096 12744 tablet_bootstrap.cc:492] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f: Bootstrap starting.
I20260928 02:18:21.401276 12744 tablet_bootstrap.cc:654] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:21.402292 12744 log.cc:824] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:21.404994 12744 tablet_bootstrap.cc:492] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f: No bootstrap required, opened a new log
I20260928 02:18:21.405268 12744 ts_tablet_manager.cc:1397] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 02:18:21.406623 12746 tablet_bootstrap.cc:492] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a: Bootstrap starting.
I20260928 02:18:21.408636 12747 tablet_bootstrap.cc:492] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8: Bootstrap starting.
I20260928 02:18:21.410341 12746 tablet_bootstrap.cc:654] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:21.411664 12746 log.cc:824] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:21.411741 12747 tablet_bootstrap.cc:654] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:21.413012 12747 log.cc:824] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:21.414794 12746 tablet_bootstrap.cc:492] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a: No bootstrap required, opened a new log
I20260928 02:18:21.415071 12746 ts_tablet_manager.cc:1397] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260928 02:18:21.415581 12747 tablet_bootstrap.cc:492] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8: No bootstrap required, opened a new log
I20260928 02:18:21.415304 12744 raft_consensus.cc:348] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } }
I20260928 02:18:21.415774 12744 raft_consensus.cc:374] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:21.415866 12747 ts_tablet_manager.cc:1397] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 02:18:21.415881 12744 raft_consensus.cc:729] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28fa0e82e3e0496c89d30fdb80df848f, State: Initialized, Role: FOLLOWER
I20260928 02:18:21.416524 12744 consensus_queue.cc:260] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } }
I20260928 02:18:21.418226 12744 ts_tablet_manager.cc:1428] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260928 02:18:21.424829 12746 raft_consensus.cc:348] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } }
I20260928 02:18:21.425334 12746 raft_consensus.cc:374] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:21.425459 12746 raft_consensus.cc:729] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f2f1dffc9a478cb5182a25c66a1a9a, State: Initialized, Role: FOLLOWER
I20260928 02:18:21.426072 12746 consensus_queue.cc:260] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } }
I20260928 02:18:21.426438 12747 raft_consensus.cc:348] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } }
I20260928 02:18:21.426760 12747 raft_consensus.cc:374] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:21.426872 12747 raft_consensus.cc:729] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86f236333f4148b886a2e7a81af6dff8, State: Initialized, Role: FOLLOWER
I20260928 02:18:21.427445 12747 consensus_queue.cc:260] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } }
I20260928 02:18:21.427698 12746 ts_tablet_manager.cc:1428] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a: Time spent starting tablet: real 0.013s user 0.008s sys 0.006s
I20260928 02:18:21.429150 12747 ts_tablet_manager.cc:1428] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
I20260928 02:18:21.531666 12751 raft_consensus.cc:482] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:18:21.532023 12751 raft_consensus.cc:504] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } }
I20260928 02:18:21.533717 12751 leader_election.cc:288] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28fa0e82e3e0496c89d30fdb80df848f (127.4.140.193:41081), 86f236333f4148b886a2e7a81af6dff8 (127.4.140.195:32857)
I20260928 02:18:21.540843 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:21.534075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:33595 (local address 127.4.140.193:41081)
0928 02:18:21.534308 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:21.534316 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:21.535440 (+ 1124us) server_negotiation.cc:407] Connection header received
0928 02:18:21.535648 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:21.535652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:21.535709 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:21.535805 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:21.536455 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.537128 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.538792 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.539628 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.539804 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:21.539918 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:21.540386 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 02:18:21.540545 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:18:21.540897 12741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:21.534171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:58299 (local address 127.4.140.195:32857)
0928 02:18:21.534388 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:21.534394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:21.535289 (+ 895us) server_negotiation.cc:407] Connection header received
0928 02:18:21.535515 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:21.535519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:21.535576 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:21.535669 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:21.536272 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.536954 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.538703 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.539560 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.539729 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:21.539811 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:21.540403 (+ 592us) server_negotiation.cc:299] Negotiation successful
0928 02:18:21.540567 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:18:21.540892 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:21.534418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:32857 (local address 127.4.140.194:58299)
0928 02:18:21.535166 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:21.535184 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:21.535366 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:21.535697 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:21.535701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:21.535724 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:21.536126 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:21.536135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.537086 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.537093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:21.538543 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:21.538561 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.539798 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.539804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:21.540114 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:21.540261 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 02:18:21.540353 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 02:18:21.540477 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":48,"thread_start_us":401,"threads_started":1}
I20260928 02:18:21.541543 12754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:21.534479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:41081 (local address 127.4.140.194:33595)
0928 02:18:21.535337 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:21.535351 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:21.535495 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:21.535825 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:21.535829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:21.535867 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:21.536286 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:21.536295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.537271 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.537276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:21.538641 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:21.538652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.539846 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.539852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:21.540151 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:21.540218 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 02:18:21.541179 (+ 961us) client_negotiation.cc:240] Negotiation successful
0928 02:18:21.541288 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":59,"thread_start_us":517,"threads_started":1}
I20260928 02:18:21.541967 12674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dadea5cb4f040d99da1afd148f8fabf" candidate_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86f236333f4148b886a2e7a81af6dff8" is_pre_election: true
I20260928 02:18:21.542526 12674 raft_consensus.cc:2454] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20f2f1dffc9a478cb5182a25c66a1a9a in term 0.
I20260928 02:18:21.542542 12395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dadea5cb4f040d99da1afd148f8fabf" candidate_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28fa0e82e3e0496c89d30fdb80df848f" is_pre_election: true
I20260928 02:18:21.543135 12395 raft_consensus.cc:2454] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20f2f1dffc9a478cb5182a25c66a1a9a in term 0.
I20260928 02:18:21.543210 12461 leader_election.cc:302] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f2f1dffc9a478cb5182a25c66a1a9a, 86f236333f4148b886a2e7a81af6dff8; no voters:
I20260928 02:18:21.543653 12751 raft_consensus.cc:2781] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:18:21.543829 12751 raft_consensus.cc:482] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:18:21.543972 12751 raft_consensus.cc:3037] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:21.546413 12751 raft_consensus.cc:504] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } }
I20260928 02:18:21.547128 12751 leader_election.cc:288] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [CANDIDATE]: Term 1 election: Requested vote from peers 28fa0e82e3e0496c89d30fdb80df848f (127.4.140.193:41081), 86f236333f4148b886a2e7a81af6dff8 (127.4.140.195:32857)
I20260928 02:18:21.547422 12395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dadea5cb4f040d99da1afd148f8fabf" candidate_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28fa0e82e3e0496c89d30fdb80df848f"
I20260928 02:18:21.547609 12395 raft_consensus.cc:3037] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:21.547621 12674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dadea5cb4f040d99da1afd148f8fabf" candidate_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86f236333f4148b886a2e7a81af6dff8"
I20260928 02:18:21.547816 12674 raft_consensus.cc:3037] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:21.549779 12395 raft_consensus.cc:2454] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f2f1dffc9a478cb5182a25c66a1a9a in term 1.
I20260928 02:18:21.549934 12674 raft_consensus.cc:2454] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f2f1dffc9a478cb5182a25c66a1a9a in term 1.
I20260928 02:18:21.550172 12464 leader_election.cc:302] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f2f1dffc9a478cb5182a25c66a1a9a, 28fa0e82e3e0496c89d30fdb80df848f; no voters:
I20260928 02:18:21.550462 12751 raft_consensus.cc:2781] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:18:21.550900 12751 raft_consensus.cc:686] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [term 1 LEADER]: Becoming Leader. State: Replica: 20f2f1dffc9a478cb5182a25c66a1a9a, State: Running, Role: LEADER
I20260928 02:18:21.551445 12751 consensus_queue.cc:237] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } }
I20260928 02:18:21.555835 12233 catalog_manager.cc:5495] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 20f2f1dffc9a478cb5182a25c66a1a9a (127.4.140.194). New cstate: current_term: 1 leader_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } health_report { overall_health: HEALTHY } } }
I20260928 02:18:21.611967 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:21.606038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55822 (local address 127.4.140.194:45669)
0928 02:18:21.606295 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:21.606303 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:21.606399 (+ 96us) server_negotiation.cc:407] Connection header received
0928 02:18:21.606735 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:21.606739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:21.606827 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:21.607008 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:18:21.607438 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.608260 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.609691 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:21.610123 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:21.610266 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:21.610726 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:18:21.611262 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:18:21.611445 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:21.611531 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 02:18:21.611618 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:18:21.620566 12674 raft_consensus.cc:1261] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8 [term 1 FOLLOWER]: Refusing update from remote peer 20f2f1dffc9a478cb5182a25c66a1a9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:18:21.620556 12395 raft_consensus.cc:1261] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f [term 1 FOLLOWER]: Refusing update from remote peer 20f2f1dffc9a478cb5182a25c66a1a9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:18:21.621645 12751 consensus_queue.cc:1035] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:18:21.621978 12755 consensus_queue.cc:1035] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:18:21.629968 12442 tablet.cc:2369] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:18:21.630921 12758 mvcc.cc:205] Tried to move back new op lower bound from 7334141549027512320 to 7334141548758642688. Current Snapshot: MvccSnapshot[applied={T|T < 7334141549027512320}]
I20260928 02:18:21.634687 12759 mvcc.cc:205] Tried to move back new op lower bound from 7334141549027512320 to 7334141548758642688. Current Snapshot: MvccSnapshot[applied={T|T < 7334141549027512320}]
I20260928 02:18:21.637769 12760 mvcc.cc:205] Tried to move back new op lower bound from 7334141549027512320 to 7334141548758642688. Current Snapshot: MvccSnapshot[applied={T|T < 7334141549027512320}]
I20260928 02:18:21.650923 12233 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:18:21.651227 12233 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:18:21.654690 12233 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c: Sending DeleteTablet for 3 replicas of tablet 4dadea5cb4f040d99da1afd148f8fabf
I20260928 02:18:21.655814 12649 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dadea5cb4f040d99da1afd148f8fabf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:21 UTC) from {username='oryx', principal='oryx/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:43898
I20260928 02:18:21.655845 12510 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dadea5cb4f040d99da1afd148f8fabf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:21 UTC) from {username='oryx', principal='oryx/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:55814
I20260928 02:18:21.656764 12233 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:18:21.657449 12233 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:18:21.658634 12371 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dadea5cb4f040d99da1afd148f8fabf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:21 UTC) from {username='oryx', principal='oryx/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:43912
I20260928 02:18:21.664335 12769 tablet_replica.cc:330] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a: stopping tablet replica
I20260928 02:18:21.664808 12769 raft_consensus.cc:2229] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:18:21.665733 12370 tablet_service.cc:1467] Processing CreateTablet for tablet d929de6da6e3466abbbaec33f341afe4 (DEFAULT_TABLE table=test-table [id=da2dc42910f64b9e932a582d167d2369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:21.665889 12648 tablet_service.cc:1467] Processing CreateTablet for tablet d929de6da6e3466abbbaec33f341afe4 (DEFAULT_TABLE table=test-table [id=da2dc42910f64b9e932a582d167d2369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:21.666296 12370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d929de6da6e3466abbbaec33f341afe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:21.666566 12648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d929de6da6e3466abbbaec33f341afe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:21.675617 12744 tablet_bootstrap.cc:492] T d929de6da6e3466abbbaec33f341afe4 P 28fa0e82e3e0496c89d30fdb80df848f: Bootstrap starting.
I20260928 02:18:21.676237 12510 tablet_service.cc:1467] Processing CreateTablet for tablet d929de6da6e3466abbbaec33f341afe4 (DEFAULT_TABLE table=test-table [id=da2dc42910f64b9e932a582d167d2369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:21.676941 12510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d929de6da6e3466abbbaec33f341afe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:21.677228 12744 tablet_bootstrap.cc:654] T d929de6da6e3466abbbaec33f341afe4 P 28fa0e82e3e0496c89d30fdb80df848f: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:21.682186 12769 raft_consensus.cc:2258] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:21.682718 12747 tablet_bootstrap.cc:492] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8: Bootstrap starting.
I20260928 02:18:21.684957 12747 tablet_bootstrap.cc:654] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:21.685272 12769 ts_tablet_manager.cc:1905] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:21.688678 12770 tablet_replica.cc:330] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f: stopping tablet replica
I20260928 02:18:21.689240 12770 raft_consensus.cc:2229] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:18:21.689927 12768 tablet_replica.cc:330] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8: stopping tablet replica
I20260928 02:18:21.689908 12770 raft_consensus.cc:2258] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:21.692345 12744 tablet_bootstrap.cc:492] T d929de6da6e3466abbbaec33f341afe4 P 28fa0e82e3e0496c89d30fdb80df848f: No bootstrap required, opened a new log
I20260928 02:18:21.692493 12744 ts_tablet_manager.cc:1397] T d929de6da6e3466abbbaec33f341afe4 P 28fa0e82e3e0496c89d30fdb80df848f: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20260928 02:18:21.693363 12744 raft_consensus.cc:348] T d929de6da6e3466abbbaec33f341afe4 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:21.693608 12744 raft_consensus.cc:374] T d929de6da6e3466abbbaec33f341afe4 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:21.693709 12744 raft_consensus.cc:729] T d929de6da6e3466abbbaec33f341afe4 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28fa0e82e3e0496c89d30fdb80df848f, State: Initialized, Role: FOLLOWER
I20260928 02:18:21.693894 12770 ts_tablet_manager.cc:1905] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:21.694049 12744 consensus_queue.cc:260] T d929de6da6e3466abbbaec33f341afe4 P 28fa0e82e3e0496c89d30fdb80df848f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:21.694525 12747 tablet_bootstrap.cc:492] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8: No bootstrap required, opened a new log
I20260928 02:18:21.694674 12747 ts_tablet_manager.cc:1397] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260928 02:18:21.694725 12744 ts_tablet_manager.cc:1428] T d929de6da6e3466abbbaec33f341afe4 P 28fa0e82e3e0496c89d30fdb80df848f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:18:21.690328 12768 raft_consensus.cc:2229] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:18:21.695319 12768 raft_consensus.cc:2258] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:21.695683 12747 raft_consensus.cc:348] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:21.695863 12747 raft_consensus.cc:374] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:21.695986 12747 raft_consensus.cc:729] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86f236333f4148b886a2e7a81af6dff8, State: Initialized, Role: FOLLOWER
I20260928 02:18:21.696316 12747 consensus_queue.cc:260] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:21.698904 12747 ts_tablet_manager.cc:1428] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 02:18:21.700481 12769 ts_tablet_manager.cc:1918] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:18:21.700667 12769 log.cc:1196] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/4dadea5cb4f040d99da1afd148f8fabf
I20260928 02:18:21.701205 12769 ts_tablet_manager.cc:1939] T 4dadea5cb4f040d99da1afd148f8fabf P 20f2f1dffc9a478cb5182a25c66a1a9a: Deleting consensus metadata
I20260928 02:18:21.702394 12746 tablet_bootstrap.cc:492] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a: Bootstrap starting.
I20260928 02:18:21.702934 12221 catalog_manager.cc:4841] TS 20f2f1dffc9a478cb5182a25c66a1a9a (127.4.140.194:45669): tablet 4dadea5cb4f040d99da1afd148f8fabf (table test-table [id=d367eca5dd6a48f3b0099e9894c4d724]) successfully deleted
I20260928 02:18:21.703310 12770 ts_tablet_manager.cc:1918] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:18:21.703444 12770 log.cc:1196] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/4dadea5cb4f040d99da1afd148f8fabf
I20260928 02:18:21.703840 12746 tablet_bootstrap.cc:654] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:21.703960 12770 ts_tablet_manager.cc:1939] T 4dadea5cb4f040d99da1afd148f8fabf P 28fa0e82e3e0496c89d30fdb80df848f: Deleting consensus metadata
I20260928 02:18:21.704653 12768 ts_tablet_manager.cc:1905] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:21.705509 12221 catalog_manager.cc:4841] TS 28fa0e82e3e0496c89d30fdb80df848f (127.4.140.193:41081): tablet 4dadea5cb4f040d99da1afd148f8fabf (table test-table [id=d367eca5dd6a48f3b0099e9894c4d724]) successfully deleted
I20260928 02:18:21.705698 12746 tablet_bootstrap.cc:492] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a: No bootstrap required, opened a new log
I20260928 02:18:21.705806 12746 ts_tablet_manager.cc:1397] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 02:18:21.706382 12746 raft_consensus.cc:348] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:21.706532 12746 raft_consensus.cc:374] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:21.706596 12746 raft_consensus.cc:729] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f2f1dffc9a478cb5182a25c66a1a9a, State: Initialized, Role: FOLLOWER
I20260928 02:18:21.706833 12746 consensus_queue.cc:260] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:21.707296 12746 ts_tablet_manager.cc:1428] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:18:21.711403 12768 ts_tablet_manager.cc:1918] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:18:21.711575 12768 log.cc:1196] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/4dadea5cb4f040d99da1afd148f8fabf
I20260928 02:18:21.712134 12768 ts_tablet_manager.cc:1939] T 4dadea5cb4f040d99da1afd148f8fabf P 86f236333f4148b886a2e7a81af6dff8: Deleting consensus metadata
I20260928 02:18:21.713876 12218 catalog_manager.cc:4841] TS 86f236333f4148b886a2e7a81af6dff8 (127.4.140.195:32857): tablet 4dadea5cb4f040d99da1afd148f8fabf (table test-table [id=d367eca5dd6a48f3b0099e9894c4d724]) successfully deleted
W20260928 02:18:21.895236 12581 tablet.cc:2369] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:18:21.925055 12720 tablet.cc:2369] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:18:22.093250 12765 raft_consensus.cc:482] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:18:22.093480 12765 raft_consensus.cc:504] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.094273 12765 leader_election.cc:288] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28fa0e82e3e0496c89d30fdb80df848f (127.4.140.193:41081), 86f236333f4148b886a2e7a81af6dff8 (127.4.140.195:32857)
I20260928 02:18:22.094830 12394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d929de6da6e3466abbbaec33f341afe4" candidate_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28fa0e82e3e0496c89d30fdb80df848f" is_pre_election: true
I20260928 02:18:22.094877 12674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d929de6da6e3466abbbaec33f341afe4" candidate_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86f236333f4148b886a2e7a81af6dff8" is_pre_election: true
I20260928 02:18:22.095084 12394 raft_consensus.cc:2454] T d929de6da6e3466abbbaec33f341afe4 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20f2f1dffc9a478cb5182a25c66a1a9a in term 0.
I20260928 02:18:22.095142 12674 raft_consensus.cc:2454] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20f2f1dffc9a478cb5182a25c66a1a9a in term 0.
I20260928 02:18:22.095547 12464 leader_election.cc:302] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20f2f1dffc9a478cb5182a25c66a1a9a, 28fa0e82e3e0496c89d30fdb80df848f, 86f236333f4148b886a2e7a81af6dff8; no voters:
I20260928 02:18:22.095871 12765 raft_consensus.cc:2781] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:18:22.096020 12765 raft_consensus.cc:482] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:18:22.096174 12765 raft_consensus.cc:3037] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:22.098238 12765 raft_consensus.cc:504] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.098773 12765 leader_election.cc:288] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [CANDIDATE]: Term 1 election: Requested vote from peers 28fa0e82e3e0496c89d30fdb80df848f (127.4.140.193:41081), 86f236333f4148b886a2e7a81af6dff8 (127.4.140.195:32857)
I20260928 02:18:22.099227 12394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d929de6da6e3466abbbaec33f341afe4" candidate_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28fa0e82e3e0496c89d30fdb80df848f"
I20260928 02:18:22.099278 12674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d929de6da6e3466abbbaec33f341afe4" candidate_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86f236333f4148b886a2e7a81af6dff8"
I20260928 02:18:22.099431 12394 raft_consensus.cc:3037] T d929de6da6e3466abbbaec33f341afe4 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:22.099464 12674 raft_consensus.cc:3037] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:22.101552 12394 raft_consensus.cc:2454] T d929de6da6e3466abbbaec33f341afe4 P 28fa0e82e3e0496c89d30fdb80df848f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f2f1dffc9a478cb5182a25c66a1a9a in term 1.
I20260928 02:18:22.101562 12674 raft_consensus.cc:2454] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f2f1dffc9a478cb5182a25c66a1a9a in term 1.
I20260928 02:18:22.101958 12461 leader_election.cc:302] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f2f1dffc9a478cb5182a25c66a1a9a, 86f236333f4148b886a2e7a81af6dff8; no voters:
I20260928 02:18:22.102241 12765 raft_consensus.cc:2781] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:18:22.102404 12765 raft_consensus.cc:686] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 1 LEADER]: Becoming Leader. State: Replica: 20f2f1dffc9a478cb5182a25c66a1a9a, State: Running, Role: LEADER
I20260928 02:18:22.102695 12765 consensus_queue.cc:237] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.105348 12234 catalog_manager.cc:5495] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 20f2f1dffc9a478cb5182a25c66a1a9a (127.4.140.194). New cstate: current_term: 1 leader_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:18:22 dist-test-slave-h37r krb5kdc[12189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561898, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.254@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM
I20260928 02:18:22.167476 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:22.154210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:44903 (local address 127.0.0.1:43550)
0928 02:18:22.154949 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:22.154967 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:22.155097 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:22.155419 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:22.155422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:22.156145 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:22.156339 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:22.156347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.157381 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.157385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:22.158341 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:22.158351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.159099 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.159104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:22.159247 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:22.160060 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:22.160079 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:22.162492 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:22.165387 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:22.165396 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:22.165402 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:22.165796 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:22.166121 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:22.166124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:22.166126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:22.166256 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:22.166709 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:22.166715 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:22.166854 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 02:18:22.166984 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 02:18:22.167193 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":417,"threads_started":1}
I20260928 02:18:22.167624 12781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:22.154299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43550 (local address 127.4.140.254:44903)
0928 02:18:22.154990 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:22.154995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:22.155021 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:18:22.155217 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:22.155220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:22.155280 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:22.155369 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:22.156465 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.157245 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.158459 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.158910 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.159002 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:22.162685 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:22.162690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:22.162694 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:22.162722 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:22.165269 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:22.165901 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:22.165905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:22.165907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:22.166028 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:22.166337 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:22.166340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:22.166342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:22.166603 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:22.166728 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:22.167059 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 02:18:22.167307 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":583,"thread_start_us":409,"threads_started":1}
I20260928 02:18:22.172891 12234 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:43550:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:18:22.173920 12234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:18:22.180591 12371 tablet_service.cc:1467] Processing CreateTablet for tablet 5f149221fe944645a23c53945b5d3835 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b988fbfb122450cada2528c3945d3b6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:18:22.180913 12510 tablet_service.cc:1467] Processing CreateTablet for tablet 5f149221fe944645a23c53945b5d3835 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b988fbfb122450cada2528c3945d3b6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:18:22.181200 12371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f149221fe944645a23c53945b5d3835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:22.181418 12510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f149221fe944645a23c53945b5d3835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:22.187217 12744 tablet_bootstrap.cc:492] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f: Bootstrap starting.
I20260928 02:18:22.188428 12649 tablet_service.cc:1467] Processing CreateTablet for tablet 5f149221fe944645a23c53945b5d3835 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b988fbfb122450cada2528c3945d3b6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:18:22.188796 12744 tablet_bootstrap.cc:654] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:22.188982 12649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f149221fe944645a23c53945b5d3835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:22.191021 12744 tablet_bootstrap.cc:492] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f: No bootstrap required, opened a new log
I20260928 02:18:22.191135 12744 ts_tablet_manager.cc:1397] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 02:18:22.191740 12744 raft_consensus.cc:348] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.192027 12744 raft_consensus.cc:374] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:22.192087 12744 raft_consensus.cc:729] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28fa0e82e3e0496c89d30fdb80df848f, State: Initialized, Role: FOLLOWER
I20260928 02:18:22.192312 12744 consensus_queue.cc:260] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.192835 12744 ts_tablet_manager.cc:1428] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:18:22.192680 12750 tablet_replica.cc:439] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } } }
I20260928 02:18:22.193010 12750 tablet_replica.cc:442] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:18:22.196969 12747 tablet_bootstrap.cc:492] T 5f149221fe944645a23c53945b5d3835 P 86f236333f4148b886a2e7a81af6dff8: Bootstrap starting.
I20260928 02:18:22.197154 12746 tablet_bootstrap.cc:492] T 5f149221fe944645a23c53945b5d3835 P 20f2f1dffc9a478cb5182a25c66a1a9a: Bootstrap starting.
I20260928 02:18:22.198855 12747 tablet_bootstrap.cc:654] T 5f149221fe944645a23c53945b5d3835 P 86f236333f4148b886a2e7a81af6dff8: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:22.198875 12746 tablet_bootstrap.cc:654] T 5f149221fe944645a23c53945b5d3835 P 20f2f1dffc9a478cb5182a25c66a1a9a: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:22.201076 12746 tablet_bootstrap.cc:492] T 5f149221fe944645a23c53945b5d3835 P 20f2f1dffc9a478cb5182a25c66a1a9a: No bootstrap required, opened a new log
I20260928 02:18:22.201257 12746 ts_tablet_manager.cc:1397] T 5f149221fe944645a23c53945b5d3835 P 20f2f1dffc9a478cb5182a25c66a1a9a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260928 02:18:22.201582 12747 tablet_bootstrap.cc:492] T 5f149221fe944645a23c53945b5d3835 P 86f236333f4148b886a2e7a81af6dff8: No bootstrap required, opened a new log
I20260928 02:18:22.201747 12747 ts_tablet_manager.cc:1397] T 5f149221fe944645a23c53945b5d3835 P 86f236333f4148b886a2e7a81af6dff8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:18:22.202090 12746 raft_consensus.cc:348] T 5f149221fe944645a23c53945b5d3835 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.202390 12746 raft_consensus.cc:374] T 5f149221fe944645a23c53945b5d3835 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:22.202488 12746 raft_consensus.cc:729] T 5f149221fe944645a23c53945b5d3835 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f2f1dffc9a478cb5182a25c66a1a9a, State: Initialized, Role: FOLLOWER
I20260928 02:18:22.202692 12747 raft_consensus.cc:348] T 5f149221fe944645a23c53945b5d3835 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.202747 12746 consensus_queue.cc:260] T 5f149221fe944645a23c53945b5d3835 P 20f2f1dffc9a478cb5182a25c66a1a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.202951 12747 raft_consensus.cc:374] T 5f149221fe944645a23c53945b5d3835 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:22.203076 12747 raft_consensus.cc:729] T 5f149221fe944645a23c53945b5d3835 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86f236333f4148b886a2e7a81af6dff8, State: Initialized, Role: FOLLOWER
I20260928 02:18:22.203219 12765 tablet_replica.cc:439] T 5f149221fe944645a23c53945b5d3835 P 20f2f1dffc9a478cb5182a25c66a1a9a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } } }
I20260928 02:18:22.203487 12746 ts_tablet_manager.cc:1428] T 5f149221fe944645a23c53945b5d3835 P 20f2f1dffc9a478cb5182a25c66a1a9a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 02:18:22.203540 12765 tablet_replica.cc:442] T 5f149221fe944645a23c53945b5d3835 P 20f2f1dffc9a478cb5182a25c66a1a9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:18:22.203431 12747 consensus_queue.cc:260] T 5f149221fe944645a23c53945b5d3835 P 86f236333f4148b886a2e7a81af6dff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.204346 12747 ts_tablet_manager.cc:1428] T 5f149221fe944645a23c53945b5d3835 P 86f236333f4148b886a2e7a81af6dff8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 02:18:22.204276 12785 tablet_replica.cc:439] T 5f149221fe944645a23c53945b5d3835 P 86f236333f4148b886a2e7a81af6dff8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } } }
I20260928 02:18:22.204605 12785 tablet_replica.cc:442] T 5f149221fe944645a23c53945b5d3835 P 86f236333f4148b886a2e7a81af6dff8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:18:22.292344 12750 raft_consensus.cc:482] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:18:22.292560 12750 raft_consensus.cc:504] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.294353 12750 leader_election.cc:288] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20f2f1dffc9a478cb5182a25c66a1a9a (127.4.140.194:45669), 86f236333f4148b886a2e7a81af6dff8 (127.4.140.195:32857)
I20260928 02:18:22.314703 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:22.294801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:39793 (local address 127.4.140.194:45669)
0928 02:18:22.308060 (+ 13259us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:22.308066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:22.308087 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:18:22.308173 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:22.308180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:22.308262 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:22.308341 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:22.308916 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.309835 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.312280 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.313515 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.313703 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:22.313778 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:22.314231 (+ 453us) server_negotiation.cc:299] Negotiation successful
0928 02:18:22.314372 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13079,"thread_start_us":12990,"threads_started":1}
I20260928 02:18:22.314725 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:22.294797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:58775 (local address 127.4.140.195:32857)
0928 02:18:22.307433 (+ 12636us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:22.307441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:22.307463 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:18:22.307685 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:22.307689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:22.307758 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:22.307875 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:22.308427 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.309451 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.312279 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.313523 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.313711 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:22.313778 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:22.314238 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 02:18:22.314373 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12074,"thread_start_us":11939,"threads_started":1}
I20260928 02:18:22.314831 12786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:22.294654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:45669 (local address 127.4.140.193:39793)
0928 02:18:22.306726 (+ 12072us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:22.306752 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:22.306898 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:22.308393 (+ 1495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:22.308401 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:22.308427 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:22.308764 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:22.308773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.309989 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.310001 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:22.312090 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:22.312105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.313728 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.313737 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:22.314003 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:22.314088 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 02:18:22.314184 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:18:22.314351 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11891,"mutex_wait_us":166,"thread_start_us":12626,"threads_started":1}
I20260928 02:18:22.314862 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:22.294655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:32857 (local address 127.4.140.193:58775)
0928 02:18:22.307364 (+ 12709us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:22.307384 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:22.307518 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:22.307919 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:22.307925 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:22.307957 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:22.308251 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:22.308261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.309634 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.309639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:22.312090 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:22.312105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.313758 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.313763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:22.314001 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:22.314087 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 02:18:22.314178 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 02:18:22.314351 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12556,"mutex_wait_us":161,"thread_start_us":12308,"threads_started":1}
I20260928 02:18:22.315515 12674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f149221fe944645a23c53945b5d3835" candidate_uuid: "28fa0e82e3e0496c89d30fdb80df848f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86f236333f4148b886a2e7a81af6dff8" is_pre_election: true
I20260928 02:18:22.315781 12674 raft_consensus.cc:2454] T 5f149221fe944645a23c53945b5d3835 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28fa0e82e3e0496c89d30fdb80df848f in term 0.
I20260928 02:18:22.315659 12535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f149221fe944645a23c53945b5d3835" candidate_uuid: "28fa0e82e3e0496c89d30fdb80df848f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" is_pre_election: true
I20260928 02:18:22.316107 12535 raft_consensus.cc:2454] T 5f149221fe944645a23c53945b5d3835 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28fa0e82e3e0496c89d30fdb80df848f in term 0.
I20260928 02:18:22.316350 12321 leader_election.cc:302] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28fa0e82e3e0496c89d30fdb80df848f, 86f236333f4148b886a2e7a81af6dff8; no voters:
I20260928 02:18:22.316776 12750 raft_consensus.cc:2781] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:18:22.316901 12750 raft_consensus.cc:482] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:18:22.316998 12750 raft_consensus.cc:3037] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:22.319092 12750 raft_consensus.cc:504] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.319726 12750 leader_election.cc:288] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f [CANDIDATE]: Term 1 election: Requested vote from peers 20f2f1dffc9a478cb5182a25c66a1a9a (127.4.140.194:45669), 86f236333f4148b886a2e7a81af6dff8 (127.4.140.195:32857)
I20260928 02:18:22.320268 12535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f149221fe944645a23c53945b5d3835" candidate_uuid: "28fa0e82e3e0496c89d30fdb80df848f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a"
I20260928 02:18:22.320254 12674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f149221fe944645a23c53945b5d3835" candidate_uuid: "28fa0e82e3e0496c89d30fdb80df848f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86f236333f4148b886a2e7a81af6dff8"
I20260928 02:18:22.320461 12535 raft_consensus.cc:3037] T 5f149221fe944645a23c53945b5d3835 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:22.320478 12674 raft_consensus.cc:3037] T 5f149221fe944645a23c53945b5d3835 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:22.323346 12674 raft_consensus.cc:2454] T 5f149221fe944645a23c53945b5d3835 P 86f236333f4148b886a2e7a81af6dff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28fa0e82e3e0496c89d30fdb80df848f in term 1.
I20260928 02:18:22.323361 12535 raft_consensus.cc:2454] T 5f149221fe944645a23c53945b5d3835 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28fa0e82e3e0496c89d30fdb80df848f in term 1.
I20260928 02:18:22.323782 12321 leader_election.cc:302] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28fa0e82e3e0496c89d30fdb80df848f, 86f236333f4148b886a2e7a81af6dff8; no voters:
I20260928 02:18:22.324132 12750 raft_consensus.cc:2781] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:18:22.324548 12750 raft_consensus.cc:686] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f [term 1 LEADER]: Becoming Leader. State: Replica: 28fa0e82e3e0496c89d30fdb80df848f, State: Running, Role: LEADER
I20260928 02:18:22.324928 12750 consensus_queue.cc:237] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.326831 12790 tablet_replica.cc:439] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f: TxnStatusTablet state changed. Reason: New leader 28fa0e82e3e0496c89d30fdb80df848f. Latest consensus state: current_term: 1 leader_uuid: "28fa0e82e3e0496c89d30fdb80df848f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } } }
I20260928 02:18:22.327229 12790 tablet_replica.cc:442] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:18:22.328001 12792 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:18:22.328210 12792 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:18:22.328642 12233 catalog_manager.cc:5495] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f reported cstate change: term changed from 0 to 1, leader changed from <none> to 28fa0e82e3e0496c89d30fdb80df848f (127.4.140.193). New cstate: current_term: 1 leader_uuid: "28fa0e82e3e0496c89d30fdb80df848f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } health_report { overall_health: UNKNOWN } } }
I20260928 02:18:22.350344 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:22.344255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:41081 (local address 127.0.0.1:43928)
0928 02:18:22.344481 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:22.344505 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:22.344668 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:22.345396 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:22.345402 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:22.345417 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:18:22.345645 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:22.345655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.346855 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.346863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:22.347823 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:22.347834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.348634 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.348639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:22.348821 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:22.348873 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:18:22.349775 (+ 902us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:18:22.349795 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 02:18:22.349882 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:18:22.350025 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 02:18:22.350409 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:22.344306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43928 (local address 127.4.140.193:41081)
0928 02:18:22.344976 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:22.344981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:22.345036 (+ 55us) server_negotiation.cc:407] Connection header received
0928 02:18:22.345141 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:22.345145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:22.345213 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:22.345363 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:18:22.345821 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.346727 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.348000 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.348483 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.348622 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:22.349104 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:18:22.349668 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:18:22.349818 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:22.349930 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 02:18:22.350006 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":356,"threads_started":1}
I20260928 02:18:22.354646 12674 raft_consensus.cc:1261] T 5f149221fe944645a23c53945b5d3835 P 86f236333f4148b886a2e7a81af6dff8 [term 1 FOLLOWER]: Refusing update from remote peer 28fa0e82e3e0496c89d30fdb80df848f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:18:22.355360 12790 consensus_queue.cc:1035] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f [LEADER]: Connected to new peer: Peer: permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:18:22.355438 12535 raft_consensus.cc:1261] T 5f149221fe944645a23c53945b5d3835 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 1 FOLLOWER]: Refusing update from remote peer 28fa0e82e3e0496c89d30fdb80df848f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:18:22.356189 12790 consensus_queue.cc:1035] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f [LEADER]: Connected to new peer: Peer: permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:18:22.360699 12785 tablet_replica.cc:439] T 5f149221fe944645a23c53945b5d3835 P 86f236333f4148b886a2e7a81af6dff8: TxnStatusTablet state changed. Reason: New leader 28fa0e82e3e0496c89d30fdb80df848f. Latest consensus state: current_term: 1 leader_uuid: "28fa0e82e3e0496c89d30fdb80df848f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } } }
I20260928 02:18:22.361079 12785 tablet_replica.cc:442] T 5f149221fe944645a23c53945b5d3835 P 86f236333f4148b886a2e7a81af6dff8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:18:22.364002 12765 tablet_replica.cc:439] T 5f149221fe944645a23c53945b5d3835 P 20f2f1dffc9a478cb5182a25c66a1a9a: TxnStatusTablet state changed. Reason: New leader 28fa0e82e3e0496c89d30fdb80df848f. Latest consensus state: current_term: 1 leader_uuid: "28fa0e82e3e0496c89d30fdb80df848f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } } }
I20260928 02:18:22.364282 12765 tablet_replica.cc:442] T 5f149221fe944645a23c53945b5d3835 P 20f2f1dffc9a478cb5182a25c66a1a9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:18:22.366357 12785 tablet_replica.cc:439] T 5f149221fe944645a23c53945b5d3835 P 86f236333f4148b886a2e7a81af6dff8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "28fa0e82e3e0496c89d30fdb80df848f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } } }
I20260928 02:18:22.366607 12785 tablet_replica.cc:442] T 5f149221fe944645a23c53945b5d3835 P 86f236333f4148b886a2e7a81af6dff8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:18:22.370796 12765 tablet_replica.cc:439] T 5f149221fe944645a23c53945b5d3835 P 20f2f1dffc9a478cb5182a25c66a1a9a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "28fa0e82e3e0496c89d30fdb80df848f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } } }
I20260928 02:18:22.371050 12765 tablet_replica.cc:442] T 5f149221fe944645a23c53945b5d3835 P 20f2f1dffc9a478cb5182a25c66a1a9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:18:22.371683 12790 tablet_replica.cc:439] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "28fa0e82e3e0496c89d30fdb80df848f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } } }
I20260928 02:18:22.371950 12790 tablet_replica.cc:442] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:18:22.374253 12790 tablet_replica.cc:439] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "28fa0e82e3e0496c89d30fdb80df848f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } } }
I20260928 02:18:22.374467 12790 tablet_replica.cc:442] T 5f149221fe944645a23c53945b5d3835 P 28fa0e82e3e0496c89d30fdb80df848f: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:18:22.389982 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:22.383886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:34951 (local address 127.4.140.193:41081)
0928 02:18:22.384112 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:22.384122 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:22.384628 (+ 506us) server_negotiation.cc:407] Connection header received
0928 02:18:22.384803 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:22.384806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:22.384868 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:22.384940 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:18:22.385573 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.386294 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.388224 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.388674 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.388767 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:22.389286 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:18:22.389379 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:18:22.389445 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:22.389607 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 02:18:22.389697 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:18:22.390162 12806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:22.383859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:41081 (local address 127.4.140.194:34951)
0928 02:18:22.384511 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:22.384531 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:22.384682 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:22.384983 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:22.384988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:22.385004 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:18:22.385363 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:22.385376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.386433 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.386439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:22.388090 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:22.388105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.388828 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.388834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:22.389022 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:22.389123 (+ 101us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:18:22.389459 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:18:22.389471 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:18:22.389561 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 02:18:22.389704 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":333,"threads_started":1}
I20260928 02:18:22.400873 12674 raft_consensus.cc:1261] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8 [term 1 FOLLOWER]: Refusing update from remote peer 20f2f1dffc9a478cb5182a25c66a1a9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:18:22.401161 12394 raft_consensus.cc:1261] T d929de6da6e3466abbbaec33f341afe4 P 28fa0e82e3e0496c89d30fdb80df848f [term 1 FOLLOWER]: Refusing update from remote peer 20f2f1dffc9a478cb5182a25c66a1a9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:18:22.401763 12755 consensus_queue.cc:1035] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:18:22.402084 12765 consensus_queue.cc:1035] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:18:22.439553 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:22.432972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:36513 (local address 127.4.140.194:45669)
0928 02:18:22.433155 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:22.433160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:22.433178 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:18:22.433234 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:22.433238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:22.433306 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:22.433391 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:18:22.434317 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.435337 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.437333 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.437962 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.438069 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:22.438753 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:18:22.438880 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:18:22.438965 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:22.439164 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 02:18:22.439245 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:18:22.440393 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:22.431757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:45669 (local address 127.4.140.193:36513)
0928 02:18:22.432564 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:22.432583 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:22.432707 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:22.433600 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:22.433605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:22.433618 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:18:22.434137 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:22.434150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.435503 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.435506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:22.437170 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:22.437184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.438274 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.438278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:22.438477 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:22.438548 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:18:22.439015 (+ 467us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:18:22.439026 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 02:18:22.439760 (+ 734us) client_negotiation.cc:240] Negotiation successful
0928 02:18:22.439953 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":393,"threads_started":1}
I20260928 02:18:22.484606 12234 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:18:22.484804 12234 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:18:22.487635 12234 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c: Sending DeleteTablet for 3 replicas of tablet d929de6da6e3466abbbaec33f341afe4
I20260928 02:18:22.488368 12371 tablet_service.cc:1514] Processing DeleteTablet for tablet d929de6da6e3466abbbaec33f341afe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:22 UTC) from {username='oryx', principal='oryx/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:43912
I20260928 02:18:22.488413 12510 tablet_service.cc:1514] Processing DeleteTablet for tablet d929de6da6e3466abbbaec33f341afe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:22 UTC) from {username='oryx', principal='oryx/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:55814
I20260928 02:18:22.488605 12649 tablet_service.cc:1514] Processing DeleteTablet for tablet d929de6da6e3466abbbaec33f341afe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:22 UTC) from {username='oryx', principal='oryx/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:43898
I20260928 02:18:22.498427 12815 tablet_replica.cc:330] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a: stopping tablet replica
I20260928 02:18:22.498648 12815 raft_consensus.cc:2229] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:18:22.499089 12815 raft_consensus.cc:2258] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:22.501521 12815 ts_tablet_manager.cc:1905] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:22.509922 12814 tablet_replica.cc:330] T d929de6da6e3466abbbaec33f341afe4 P 28fa0e82e3e0496c89d30fdb80df848f: stopping tablet replica
I20260928 02:18:22.510439 12816 tablet_replica.cc:330] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8: stopping tablet replica
I20260928 02:18:22.510763 12816 raft_consensus.cc:2229] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:18:22.511085 12814 raft_consensus.cc:2229] T d929de6da6e3466abbbaec33f341afe4 P 28fa0e82e3e0496c89d30fdb80df848f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:18:22.511404 12816 raft_consensus.cc:2258] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:22.511631 12815 ts_tablet_manager.cc:1918] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:18:22.511773 12815 log.cc:1196] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/d929de6da6e3466abbbaec33f341afe4
I20260928 02:18:22.511705 12814 raft_consensus.cc:2258] T d929de6da6e3466abbbaec33f341afe4 P 28fa0e82e3e0496c89d30fdb80df848f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:22.512276 12815 ts_tablet_manager.cc:1939] T d929de6da6e3466abbbaec33f341afe4 P 20f2f1dffc9a478cb5182a25c66a1a9a: Deleting consensus metadata
I20260928 02:18:22.513299 12816 ts_tablet_manager.cc:1905] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:22.513433 12221 catalog_manager.cc:4841] TS 20f2f1dffc9a478cb5182a25c66a1a9a (127.4.140.194:45669): tablet d929de6da6e3466abbbaec33f341afe4 (table test-table [id=da2dc42910f64b9e932a582d167d2369]) successfully deleted
I20260928 02:18:22.514421 12814 ts_tablet_manager.cc:1905] T d929de6da6e3466abbbaec33f341afe4 P 28fa0e82e3e0496c89d30fdb80df848f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:22.522599 12814 ts_tablet_manager.cc:1918] T d929de6da6e3466abbbaec33f341afe4 P 28fa0e82e3e0496c89d30fdb80df848f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:18:22.522763 12814 log.cc:1196] T d929de6da6e3466abbbaec33f341afe4 P 28fa0e82e3e0496c89d30fdb80df848f: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/d929de6da6e3466abbbaec33f341afe4
I20260928 02:18:22.523192 12816 ts_tablet_manager.cc:1918] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:18:22.523294 12814 ts_tablet_manager.cc:1939] T d929de6da6e3466abbbaec33f341afe4 P 28fa0e82e3e0496c89d30fdb80df848f: Deleting consensus metadata
I20260928 02:18:22.523415 12816 log.cc:1196] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/d929de6da6e3466abbbaec33f341afe4
I20260928 02:18:22.523928 12816 ts_tablet_manager.cc:1939] T d929de6da6e3466abbbaec33f341afe4 P 86f236333f4148b886a2e7a81af6dff8: Deleting consensus metadata
I20260928 02:18:22.524542 12221 catalog_manager.cc:4841] TS 28fa0e82e3e0496c89d30fdb80df848f (127.4.140.193:41081): tablet d929de6da6e3466abbbaec33f341afe4 (table test-table [id=da2dc42910f64b9e932a582d167d2369]) successfully deleted
I20260928 02:18:22.524991 12218 catalog_manager.cc:4841] TS 86f236333f4148b886a2e7a81af6dff8 (127.4.140.195:32857): tablet d929de6da6e3466abbbaec33f341afe4 (table test-table [id=da2dc42910f64b9e932a582d167d2369]) successfully deleted
Sep 28 02:18:22 dist-test-slave-h37r krb5kdc[12189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:18:22 dist-test-slave-h37r krb5kdc[12189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:22 dist-test-slave-h37r krb5kdc[12189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:18:22.559202 12781 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:22.547717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43556 (local address 127.4.140.254:44903)
0928 02:18:22.547961 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:22.547968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:22.548356 (+ 388us) server_negotiation.cc:407] Connection header received
0928 02:18:22.548505 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:22.548511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:22.548594 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:22.548718 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:22.550021 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.551155 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.552548 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.553222 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.553347 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:22.558960 (+ 5613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43556: BlockingRecv error: recv got EOF from 127.0.0.1:43556 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 02:18:22 dist-test-slave-h37r krb5kdc[12189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM
I20260928 02:18:22.578032 12781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:22.562369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43568 (local address 127.4.140.254:44903)
0928 02:18:22.562578 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:22.562583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:22.562798 (+ 215us) server_negotiation.cc:407] Connection header received
0928 02:18:22.563025 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:22.563029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:22.563100 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:22.563190 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:22.564795 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.565568 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.567413 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.567856 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.568360 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:22.572746 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:22.572751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:22.572755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:22.572789 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:22.575310 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:22.576107 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:22.576112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:22.576116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:22.576257 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:22.576656 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:22.576661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:22.576663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:22.576918 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:22.577203 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:22.577463 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 02:18:22.577696 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:18:22.581704 12234 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:18:22.582634 12234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:18:22.589811 12371 tablet_service.cc:1467] Processing CreateTablet for tablet 24fd1f79e6c24ea9bab0b77859511191 (DEFAULT_TABLE table=test-table [id=c7d2ea74957b432f9eeb64e69aab08da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:22.590032 12510 tablet_service.cc:1467] Processing CreateTablet for tablet 24fd1f79e6c24ea9bab0b77859511191 (DEFAULT_TABLE table=test-table [id=c7d2ea74957b432f9eeb64e69aab08da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:22.590327 12371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24fd1f79e6c24ea9bab0b77859511191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:22.590525 12510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24fd1f79e6c24ea9bab0b77859511191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:22.595723 12746 tablet_bootstrap.cc:492] T 24fd1f79e6c24ea9bab0b77859511191 P 20f2f1dffc9a478cb5182a25c66a1a9a: Bootstrap starting.
I20260928 02:18:22.596956 12649 tablet_service.cc:1467] Processing CreateTablet for tablet 24fd1f79e6c24ea9bab0b77859511191 (DEFAULT_TABLE table=test-table [id=c7d2ea74957b432f9eeb64e69aab08da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:22.597400 12649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24fd1f79e6c24ea9bab0b77859511191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:22.597472 12746 tablet_bootstrap.cc:654] T 24fd1f79e6c24ea9bab0b77859511191 P 20f2f1dffc9a478cb5182a25c66a1a9a: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:22.599423 12746 tablet_bootstrap.cc:492] T 24fd1f79e6c24ea9bab0b77859511191 P 20f2f1dffc9a478cb5182a25c66a1a9a: No bootstrap required, opened a new log
I20260928 02:18:22.599602 12746 ts_tablet_manager.cc:1397] T 24fd1f79e6c24ea9bab0b77859511191 P 20f2f1dffc9a478cb5182a25c66a1a9a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 02:18:22.600436 12746 raft_consensus.cc:348] T 24fd1f79e6c24ea9bab0b77859511191 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.600718 12746 raft_consensus.cc:374] T 24fd1f79e6c24ea9bab0b77859511191 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:22.600812 12746 raft_consensus.cc:729] T 24fd1f79e6c24ea9bab0b77859511191 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f2f1dffc9a478cb5182a25c66a1a9a, State: Initialized, Role: FOLLOWER
I20260928 02:18:22.601037 12746 consensus_queue.cc:260] T 24fd1f79e6c24ea9bab0b77859511191 P 20f2f1dffc9a478cb5182a25c66a1a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.602092 12744 tablet_bootstrap.cc:492] T 24fd1f79e6c24ea9bab0b77859511191 P 28fa0e82e3e0496c89d30fdb80df848f: Bootstrap starting.
I20260928 02:18:22.602231 12746 ts_tablet_manager.cc:1428] T 24fd1f79e6c24ea9bab0b77859511191 P 20f2f1dffc9a478cb5182a25c66a1a9a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:18:22.603701 12744 tablet_bootstrap.cc:654] T 24fd1f79e6c24ea9bab0b77859511191 P 28fa0e82e3e0496c89d30fdb80df848f: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:22.606691 12747 tablet_bootstrap.cc:492] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8: Bootstrap starting.
I20260928 02:18:22.608358 12747 tablet_bootstrap.cc:654] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:22.610769 12747 tablet_bootstrap.cc:492] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8: No bootstrap required, opened a new log
I20260928 02:18:22.610924 12747 ts_tablet_manager.cc:1397] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260928 02:18:22.611735 12747 raft_consensus.cc:348] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.612093 12747 raft_consensus.cc:374] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:22.612205 12747 raft_consensus.cc:729] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86f236333f4148b886a2e7a81af6dff8, State: Initialized, Role: FOLLOWER
I20260928 02:18:22.612448 12747 consensus_queue.cc:260] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.613191 12747 ts_tablet_manager.cc:1428] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:18:22.618435 12744 tablet_bootstrap.cc:492] T 24fd1f79e6c24ea9bab0b77859511191 P 28fa0e82e3e0496c89d30fdb80df848f: No bootstrap required, opened a new log
I20260928 02:18:22.618628 12744 ts_tablet_manager.cc:1397] T 24fd1f79e6c24ea9bab0b77859511191 P 28fa0e82e3e0496c89d30fdb80df848f: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.006s
I20260928 02:18:22.619306 12744 raft_consensus.cc:348] T 24fd1f79e6c24ea9bab0b77859511191 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.619534 12744 raft_consensus.cc:374] T 24fd1f79e6c24ea9bab0b77859511191 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:22.619668 12744 raft_consensus.cc:729] T 24fd1f79e6c24ea9bab0b77859511191 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28fa0e82e3e0496c89d30fdb80df848f, State: Initialized, Role: FOLLOWER
I20260928 02:18:22.619947 12744 consensus_queue.cc:260] T 24fd1f79e6c24ea9bab0b77859511191 P 28fa0e82e3e0496c89d30fdb80df848f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.620447 12744 ts_tablet_manager.cc:1428] T 24fd1f79e6c24ea9bab0b77859511191 P 28fa0e82e3e0496c89d30fdb80df848f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:18:22.771466 12785 raft_consensus.cc:482] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:18:22.771674 12785 raft_consensus.cc:504] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.773234 12785 leader_election.cc:288] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28fa0e82e3e0496c89d30fdb80df848f (127.4.140.193:41081), 20f2f1dffc9a478cb5182a25c66a1a9a (127.4.140.194:45669)
I20260928 02:18:22.780699 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:22.773519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:59369 (local address 127.4.140.193:41081)
0928 02:18:22.773729 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:22.773737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:22.774065 (+ 328us) server_negotiation.cc:407] Connection header received
0928 02:18:22.774401 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:22.774410 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:22.774515 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:22.774599 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:22.775393 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.776696 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.778628 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.779576 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.779738 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:22.779813 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:22.780229 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 02:18:22.780383 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:18:22.781073 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:22.773427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:41081 (local address 127.4.140.195:59369)
0928 02:18:22.773954 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:22.773975 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:22.774181 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:22.774830 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:22.774833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:22.774854 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:22.775224 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:22.775252 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.776847 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.776854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:22.778471 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:22.778483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.779802 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.779808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:22.780030 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:22.780073 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:18:22.780187 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:18:22.780341 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":78,"thread_start_us":293,"threads_started":1}
I20260928 02:18:22.781888 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:22.774122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:45669 (local address 127.4.140.195:49111)
0928 02:18:22.774975 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:22.774997 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:22.775153 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:22.775548 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:22.775552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:22.775574 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:22.775886 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:22.775940 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.776921 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.776927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:22.779302 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:22.779319 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.781078 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.781083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:22.781276 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:22.781306 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 02:18:22.781393 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:18:22.781519 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":1,"thread_start_us":480,"threads_started":1}
I20260928 02:18:22.781934 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:22.774265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:49111 (local address 127.4.140.194:45669)
0928 02:18:22.774440 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:22.774449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:22.775114 (+ 665us) server_negotiation.cc:407] Connection header received
0928 02:18:22.775338 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:22.775347 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:22.775424 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:22.775524 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:22.776111 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.776791 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.779497 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.780649 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.780815 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:22.780921 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:22.781455 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 02:18:22.781613 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:18:22.782081 12395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24fd1f79e6c24ea9bab0b77859511191" candidate_uuid: "86f236333f4148b886a2e7a81af6dff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28fa0e82e3e0496c89d30fdb80df848f" is_pre_election: true
I20260928 02:18:22.782390 12395 raft_consensus.cc:2454] T 24fd1f79e6c24ea9bab0b77859511191 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86f236333f4148b886a2e7a81af6dff8 in term 0.
I20260928 02:18:22.782866 12535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24fd1f79e6c24ea9bab0b77859511191" candidate_uuid: "86f236333f4148b886a2e7a81af6dff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" is_pre_election: true
I20260928 02:18:22.782985 12603 leader_election.cc:302] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28fa0e82e3e0496c89d30fdb80df848f, 86f236333f4148b886a2e7a81af6dff8; no voters:
I20260928 02:18:22.783152 12535 raft_consensus.cc:2454] T 24fd1f79e6c24ea9bab0b77859511191 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86f236333f4148b886a2e7a81af6dff8 in term 0.
I20260928 02:18:22.783406 12785 raft_consensus.cc:2781] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:18:22.783531 12785 raft_consensus.cc:482] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:18:22.783623 12785 raft_consensus.cc:3037] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:22.785709 12785 raft_consensus.cc:504] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.786201 12785 leader_election.cc:288] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [CANDIDATE]: Term 1 election: Requested vote from peers 28fa0e82e3e0496c89d30fdb80df848f (127.4.140.193:41081), 20f2f1dffc9a478cb5182a25c66a1a9a (127.4.140.194:45669)
I20260928 02:18:22.786779 12535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24fd1f79e6c24ea9bab0b77859511191" candidate_uuid: "86f236333f4148b886a2e7a81af6dff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a"
I20260928 02:18:22.786760 12395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24fd1f79e6c24ea9bab0b77859511191" candidate_uuid: "86f236333f4148b886a2e7a81af6dff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28fa0e82e3e0496c89d30fdb80df848f"
I20260928 02:18:22.786980 12535 raft_consensus.cc:3037] T 24fd1f79e6c24ea9bab0b77859511191 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:22.786986 12395 raft_consensus.cc:3037] T 24fd1f79e6c24ea9bab0b77859511191 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:22.790024 12395 raft_consensus.cc:2454] T 24fd1f79e6c24ea9bab0b77859511191 P 28fa0e82e3e0496c89d30fdb80df848f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86f236333f4148b886a2e7a81af6dff8 in term 1.
I20260928 02:18:22.790017 12535 raft_consensus.cc:2454] T 24fd1f79e6c24ea9bab0b77859511191 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86f236333f4148b886a2e7a81af6dff8 in term 1.
I20260928 02:18:22.790441 12603 leader_election.cc:302] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f2f1dffc9a478cb5182a25c66a1a9a, 86f236333f4148b886a2e7a81af6dff8; no voters:
I20260928 02:18:22.790808 12785 raft_consensus.cc:2781] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:18:22.791316 12785 raft_consensus.cc:686] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [term 1 LEADER]: Becoming Leader. State: Replica: 86f236333f4148b886a2e7a81af6dff8, State: Running, Role: LEADER
I20260928 02:18:22.791764 12785 consensus_queue.cc:237] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.795717 12232 catalog_manager.cc:5495] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86f236333f4148b886a2e7a81af6dff8 (127.4.140.195). New cstate: current_term: 1 leader_uuid: "86f236333f4148b886a2e7a81af6dff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } health_report { overall_health: HEALTHY } } }
I20260928 02:18:22.853785 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:22.848332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43908 (local address 127.4.140.195:32857)
0928 02:18:22.848767 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:22.848772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:22.848788 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:18:22.848873 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:22.848876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:22.848946 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:22.849093 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:18:22.849406 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.850158 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.851589 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:22.852063 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:22.852197 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:22.852636 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:18:22.853188 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:18:22.853370 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:22.853418 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 02:18:22.853494 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":230,"threads_started":1}
I20260928 02:18:22.857784 12535 raft_consensus.cc:1261] T 24fd1f79e6c24ea9bab0b77859511191 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 1 FOLLOWER]: Refusing update from remote peer 86f236333f4148b886a2e7a81af6dff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:18:22.857872 12395 raft_consensus.cc:1261] T 24fd1f79e6c24ea9bab0b77859511191 P 28fa0e82e3e0496c89d30fdb80df848f [term 1 FOLLOWER]: Refusing update from remote peer 86f236333f4148b886a2e7a81af6dff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:18:22.858551 12834 consensus_queue.cc:1035] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:18:22.858970 12785 consensus_queue.cc:1035] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:18:22.880589 12234 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:18:22.880873 12234 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:18:22.884454 12234 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c: Sending DeleteTablet for 3 replicas of tablet 24fd1f79e6c24ea9bab0b77859511191
I20260928 02:18:22.885373 12371 tablet_service.cc:1514] Processing DeleteTablet for tablet 24fd1f79e6c24ea9bab0b77859511191 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:22 UTC) from {username='oryx', principal='oryx/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:43912
I20260928 02:18:22.885687 12814 tablet_replica.cc:330] T 24fd1f79e6c24ea9bab0b77859511191 P 28fa0e82e3e0496c89d30fdb80df848f: stopping tablet replica
I20260928 02:18:22.885900 12814 raft_consensus.cc:2229] T 24fd1f79e6c24ea9bab0b77859511191 P 28fa0e82e3e0496c89d30fdb80df848f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:18:22.886147 12814 raft_consensus.cc:2258] T 24fd1f79e6c24ea9bab0b77859511191 P 28fa0e82e3e0496c89d30fdb80df848f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:22.886994 12232 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:18:22.887809 12232 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:18:22.888341 12510 tablet_service.cc:1514] Processing DeleteTablet for tablet 24fd1f79e6c24ea9bab0b77859511191 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:22 UTC) from {username='oryx', principal='oryx/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:55814
I20260928 02:18:22.888342 12649 tablet_service.cc:1514] Processing DeleteTablet for tablet 24fd1f79e6c24ea9bab0b77859511191 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:22 UTC) from {username='oryx', principal='oryx/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:43898
I20260928 02:18:22.888665 12815 tablet_replica.cc:330] T 24fd1f79e6c24ea9bab0b77859511191 P 20f2f1dffc9a478cb5182a25c66a1a9a: stopping tablet replica
I20260928 02:18:22.888866 12815 raft_consensus.cc:2229] T 24fd1f79e6c24ea9bab0b77859511191 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:18:22.889098 12815 raft_consensus.cc:2258] T 24fd1f79e6c24ea9bab0b77859511191 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:22.889416 12816 tablet_replica.cc:330] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8: stopping tablet replica
I20260928 02:18:22.889642 12816 raft_consensus.cc:2229] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:18:22.889698 12814 ts_tablet_manager.cc:1905] T 24fd1f79e6c24ea9bab0b77859511191 P 28fa0e82e3e0496c89d30fdb80df848f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:22.890179 12816 raft_consensus.cc:2258] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:22.891670 12816 ts_tablet_manager.cc:1905] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:22.898301 12814 ts_tablet_manager.cc:1918] T 24fd1f79e6c24ea9bab0b77859511191 P 28fa0e82e3e0496c89d30fdb80df848f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:18:22.898389 12816 ts_tablet_manager.cc:1918] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:18:22.898455 12814 log.cc:1196] T 24fd1f79e6c24ea9bab0b77859511191 P 28fa0e82e3e0496c89d30fdb80df848f: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/24fd1f79e6c24ea9bab0b77859511191
I20260928 02:18:22.898522 12816 log.cc:1196] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/24fd1f79e6c24ea9bab0b77859511191
I20260928 02:18:22.898896 12814 ts_tablet_manager.cc:1939] T 24fd1f79e6c24ea9bab0b77859511191 P 28fa0e82e3e0496c89d30fdb80df848f: Deleting consensus metadata
I20260928 02:18:22.898909 12816 ts_tablet_manager.cc:1939] T 24fd1f79e6c24ea9bab0b77859511191 P 86f236333f4148b886a2e7a81af6dff8: Deleting consensus metadata
I20260928 02:18:22.899973 12221 catalog_manager.cc:4841] TS 28fa0e82e3e0496c89d30fdb80df848f (127.4.140.193:41081): tablet 24fd1f79e6c24ea9bab0b77859511191 (table test-table [id=c7d2ea74957b432f9eeb64e69aab08da]) successfully deleted
I20260928 02:18:22.899950 12218 catalog_manager.cc:4841] TS 86f236333f4148b886a2e7a81af6dff8 (127.4.140.195:32857): tablet 24fd1f79e6c24ea9bab0b77859511191 (table test-table [id=c7d2ea74957b432f9eeb64e69aab08da]) successfully deleted
I20260928 02:18:22.901597 12371 tablet_service.cc:1467] Processing CreateTablet for tablet f697174a08a24bf6b4a77f69ed0062f6 (DEFAULT_TABLE table=test-table [id=e38948a3b619478da6df96811c39606f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:22.902207 12371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f697174a08a24bf6b4a77f69ed0062f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:22.902642 12815 ts_tablet_manager.cc:1905] T 24fd1f79e6c24ea9bab0b77859511191 P 20f2f1dffc9a478cb5182a25c66a1a9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:22.904611 12510 tablet_service.cc:1467] Processing CreateTablet for tablet f697174a08a24bf6b4a77f69ed0062f6 (DEFAULT_TABLE table=test-table [id=e38948a3b619478da6df96811c39606f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:22.905086 12510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f697174a08a24bf6b4a77f69ed0062f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:22.908871 12649 tablet_service.cc:1467] Processing CreateTablet for tablet f697174a08a24bf6b4a77f69ed0062f6 (DEFAULT_TABLE table=test-table [id=e38948a3b619478da6df96811c39606f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:22.909325 12649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f697174a08a24bf6b4a77f69ed0062f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:22.909718 12744 tablet_bootstrap.cc:492] T f697174a08a24bf6b4a77f69ed0062f6 P 28fa0e82e3e0496c89d30fdb80df848f: Bootstrap starting.
I20260928 02:18:22.911382 12744 tablet_bootstrap.cc:654] T f697174a08a24bf6b4a77f69ed0062f6 P 28fa0e82e3e0496c89d30fdb80df848f: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:22.922892 12815 ts_tablet_manager.cc:1918] T 24fd1f79e6c24ea9bab0b77859511191 P 20f2f1dffc9a478cb5182a25c66a1a9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:18:22.925196 12746 tablet_bootstrap.cc:492] T f697174a08a24bf6b4a77f69ed0062f6 P 20f2f1dffc9a478cb5182a25c66a1a9a: Bootstrap starting.
I20260928 02:18:22.925541 12815 log.cc:1196] T 24fd1f79e6c24ea9bab0b77859511191 P 20f2f1dffc9a478cb5182a25c66a1a9a: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/24fd1f79e6c24ea9bab0b77859511191
I20260928 02:18:22.926162 12815 ts_tablet_manager.cc:1939] T 24fd1f79e6c24ea9bab0b77859511191 P 20f2f1dffc9a478cb5182a25c66a1a9a: Deleting consensus metadata
I20260928 02:18:22.926487 12747 tablet_bootstrap.cc:492] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8: Bootstrap starting.
I20260928 02:18:22.928642 12747 tablet_bootstrap.cc:654] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:22.931485 12747 tablet_bootstrap.cc:492] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8: No bootstrap required, opened a new log
I20260928 02:18:22.931793 12747 ts_tablet_manager.cc:1397] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:18:22.932765 12747 raft_consensus.cc:348] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.933102 12747 raft_consensus.cc:374] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:22.933229 12747 raft_consensus.cc:729] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86f236333f4148b886a2e7a81af6dff8, State: Initialized, Role: FOLLOWER
I20260928 02:18:22.933585 12747 consensus_queue.cc:260] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.937678 12747 ts_tablet_manager.cc:1428] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260928 02:18:22.927071 12746 tablet_bootstrap.cc:654] T f697174a08a24bf6b4a77f69ed0062f6 P 20f2f1dffc9a478cb5182a25c66a1a9a: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:22.942713 12221 catalog_manager.cc:4841] TS 20f2f1dffc9a478cb5182a25c66a1a9a (127.4.140.194:45669): tablet 24fd1f79e6c24ea9bab0b77859511191 (table test-table [id=c7d2ea74957b432f9eeb64e69aab08da]) successfully deleted
I20260928 02:18:22.944423 12744 tablet_bootstrap.cc:492] T f697174a08a24bf6b4a77f69ed0062f6 P 28fa0e82e3e0496c89d30fdb80df848f: No bootstrap required, opened a new log
I20260928 02:18:22.944591 12744 ts_tablet_manager.cc:1397] T f697174a08a24bf6b4a77f69ed0062f6 P 28fa0e82e3e0496c89d30fdb80df848f: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.018s
I20260928 02:18:22.945174 12746 tablet_bootstrap.cc:492] T f697174a08a24bf6b4a77f69ed0062f6 P 20f2f1dffc9a478cb5182a25c66a1a9a: No bootstrap required, opened a new log
I20260928 02:18:22.945363 12746 ts_tablet_manager.cc:1397] T f697174a08a24bf6b4a77f69ed0062f6 P 20f2f1dffc9a478cb5182a25c66a1a9a: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20260928 02:18:22.945323 12744 raft_consensus.cc:348] T f697174a08a24bf6b4a77f69ed0062f6 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.945622 12744 raft_consensus.cc:374] T f697174a08a24bf6b4a77f69ed0062f6 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:22.945703 12744 raft_consensus.cc:729] T f697174a08a24bf6b4a77f69ed0062f6 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28fa0e82e3e0496c89d30fdb80df848f, State: Initialized, Role: FOLLOWER
I20260928 02:18:22.945966 12744 consensus_queue.cc:260] T f697174a08a24bf6b4a77f69ed0062f6 P 28fa0e82e3e0496c89d30fdb80df848f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.946149 12746 raft_consensus.cc:348] T f697174a08a24bf6b4a77f69ed0062f6 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.946403 12746 raft_consensus.cc:374] T f697174a08a24bf6b4a77f69ed0062f6 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:22.946468 12746 raft_consensus.cc:729] T f697174a08a24bf6b4a77f69ed0062f6 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f2f1dffc9a478cb5182a25c66a1a9a, State: Initialized, Role: FOLLOWER
I20260928 02:18:22.946540 12744 ts_tablet_manager.cc:1428] T f697174a08a24bf6b4a77f69ed0062f6 P 28fa0e82e3e0496c89d30fdb80df848f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:18:22.946718 12746 consensus_queue.cc:260] T f697174a08a24bf6b4a77f69ed0062f6 P 20f2f1dffc9a478cb5182a25c66a1a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:22.947263 12746 ts_tablet_manager.cc:1428] T f697174a08a24bf6b4a77f69ed0062f6 P 20f2f1dffc9a478cb5182a25c66a1a9a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:18:23.023093 12840 raft_consensus.cc:482] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:18:23.023347 12840 raft_consensus.cc:504] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:23.024296 12840 leader_election.cc:288] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28fa0e82e3e0496c89d30fdb80df848f (127.4.140.193:41081), 20f2f1dffc9a478cb5182a25c66a1a9a (127.4.140.194:45669)
I20260928 02:18:23.024910 12535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f697174a08a24bf6b4a77f69ed0062f6" candidate_uuid: "86f236333f4148b886a2e7a81af6dff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" is_pre_election: true
I20260928 02:18:23.024938 12395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f697174a08a24bf6b4a77f69ed0062f6" candidate_uuid: "86f236333f4148b886a2e7a81af6dff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28fa0e82e3e0496c89d30fdb80df848f" is_pre_election: true
I20260928 02:18:23.025166 12535 raft_consensus.cc:2454] T f697174a08a24bf6b4a77f69ed0062f6 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86f236333f4148b886a2e7a81af6dff8 in term 0.
I20260928 02:18:23.025177 12395 raft_consensus.cc:2454] T f697174a08a24bf6b4a77f69ed0062f6 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86f236333f4148b886a2e7a81af6dff8 in term 0.
I20260928 02:18:23.025642 12603 leader_election.cc:302] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f2f1dffc9a478cb5182a25c66a1a9a, 86f236333f4148b886a2e7a81af6dff8; no voters:
I20260928 02:18:23.025934 12840 raft_consensus.cc:2781] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:18:23.026073 12840 raft_consensus.cc:482] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:18:23.026175 12840 raft_consensus.cc:3037] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:23.028640 12840 raft_consensus.cc:504] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:23.029297 12840 leader_election.cc:288] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [CANDIDATE]: Term 1 election: Requested vote from peers 28fa0e82e3e0496c89d30fdb80df848f (127.4.140.193:41081), 20f2f1dffc9a478cb5182a25c66a1a9a (127.4.140.194:45669)
I20260928 02:18:23.029717 12395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f697174a08a24bf6b4a77f69ed0062f6" candidate_uuid: "86f236333f4148b886a2e7a81af6dff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28fa0e82e3e0496c89d30fdb80df848f"
I20260928 02:18:23.029843 12535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f697174a08a24bf6b4a77f69ed0062f6" candidate_uuid: "86f236333f4148b886a2e7a81af6dff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a"
I20260928 02:18:23.029935 12395 raft_consensus.cc:3037] T f697174a08a24bf6b4a77f69ed0062f6 P 28fa0e82e3e0496c89d30fdb80df848f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:23.030056 12535 raft_consensus.cc:3037] T f697174a08a24bf6b4a77f69ed0062f6 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:23.032001 12395 raft_consensus.cc:2454] T f697174a08a24bf6b4a77f69ed0062f6 P 28fa0e82e3e0496c89d30fdb80df848f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86f236333f4148b886a2e7a81af6dff8 in term 1.
I20260928 02:18:23.032428 12603 leader_election.cc:302] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28fa0e82e3e0496c89d30fdb80df848f, 86f236333f4148b886a2e7a81af6dff8; no voters:
I20260928 02:18:23.032693 12840 raft_consensus.cc:2781] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:18:23.032913 12840 raft_consensus.cc:686] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [term 1 LEADER]: Becoming Leader. State: Replica: 86f236333f4148b886a2e7a81af6dff8, State: Running, Role: LEADER
I20260928 02:18:23.033053 12535 raft_consensus.cc:2454] T f697174a08a24bf6b4a77f69ed0062f6 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86f236333f4148b886a2e7a81af6dff8 in term 1.
I20260928 02:18:23.033309 12840 consensus_queue.cc:237] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } }
I20260928 02:18:23.037565 12234 catalog_manager.cc:5495] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86f236333f4148b886a2e7a81af6dff8 (127.4.140.195). New cstate: current_term: 1 leader_uuid: "86f236333f4148b886a2e7a81af6dff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86f236333f4148b886a2e7a81af6dff8" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 32857 } health_report { overall_health: HEALTHY } } }
I20260928 02:18:23.092700 12850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:23.086215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:41081 (local address 127.4.140.195:59143)
0928 02:18:23.086907 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:23.086927 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:23.087064 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:23.087396 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:23.087399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:23.087408 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:18:23.087708 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:23.087715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:23.089130 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:23.089133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:23.090571 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:23.090584 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:23.091522 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:23.091527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:23.091662 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:23.091724 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:18:23.092187 (+ 463us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:18:23.092196 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:18:23.092268 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:18:23.092366 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":373,"threads_started":1}
I20260928 02:18:23.092690 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:23.086204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:59143 (local address 127.4.140.193:41081)
0928 02:18:23.086391 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:23.086398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:23.087014 (+ 616us) server_negotiation.cc:407] Connection header received
0928 02:18:23.087193 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:23.087199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:23.087282 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:23.087381 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:18:23.087865 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:23.089002 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:23.090711 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:23.091352 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:23.091482 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:23.091938 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:18:23.092083 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:18:23.092168 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:23.092314 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 02:18:23.092395 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:18:23.101787 12535 raft_consensus.cc:1261] T f697174a08a24bf6b4a77f69ed0062f6 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 1 FOLLOWER]: Refusing update from remote peer 86f236333f4148b886a2e7a81af6dff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:18:23.101787 12395 raft_consensus.cc:1261] T f697174a08a24bf6b4a77f69ed0062f6 P 28fa0e82e3e0496c89d30fdb80df848f [term 1 FOLLOWER]: Refusing update from remote peer 86f236333f4148b886a2e7a81af6dff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:18:23.102347 12840 consensus_queue.cc:1035] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20f2f1dffc9a478cb5182a25c66a1a9a" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 45669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:18:23.102696 12785 consensus_queue.cc:1035] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28fa0e82e3e0496c89d30fdb80df848f" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:18:23.148849 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:23.141352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:45995 (local address 127.4.140.195:32857)
0928 02:18:23.141508 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:23.141514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:23.141581 (+ 67us) server_negotiation.cc:407] Connection header received
0928 02:18:23.141822 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:23.141830 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:23.141940 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:23.142039 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:18:23.142799 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:23.144366 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:23.147172 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:23.147604 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:23.147745 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:23.148170 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:18:23.148273 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:18:23.148378 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:23.148482 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 02:18:23.148553 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:18:23.149048 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:23.140825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:32857 (local address 127.4.140.193:45995)
0928 02:18:23.141483 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:23.141502 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:23.141688 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:23.142256 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:23.142264 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:23.142282 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:18:23.142613 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:23.142627 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:23.145543 (+ 2916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:23.145548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:23.147027 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:23.147038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:23.147754 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:23.147760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:23.147969 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:23.148018 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:18:23.148379 (+ 361us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:18:23.148391 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:18:23.148464 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:18:23.148595 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":602,"threads_started":1}
I20260928 02:18:23.204707 12232 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:18:23.204973 12232 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:18:23.207847 12232 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72974f12c2e140c1aeb70bd3d355c05c: Sending DeleteTablet for 3 replicas of tablet f697174a08a24bf6b4a77f69ed0062f6
I20260928 02:18:23.208775 12510 tablet_service.cc:1514] Processing DeleteTablet for tablet f697174a08a24bf6b4a77f69ed0062f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:23 UTC) from {username='oryx', principal='oryx/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:55814
I20260928 02:18:23.209105 12815 tablet_replica.cc:330] T f697174a08a24bf6b4a77f69ed0062f6 P 20f2f1dffc9a478cb5182a25c66a1a9a: stopping tablet replica
I20260928 02:18:23.209300 12815 raft_consensus.cc:2229] T f697174a08a24bf6b4a77f69ed0062f6 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:18:23.209522 12815 raft_consensus.cc:2258] T f697174a08a24bf6b4a77f69ed0062f6 P 20f2f1dffc9a478cb5182a25c66a1a9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:23.209839 12371 tablet_service.cc:1514] Processing DeleteTablet for tablet f697174a08a24bf6b4a77f69ed0062f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:23 UTC) from {username='oryx', principal='oryx/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:43912
I20260928 02:18:23.210134 12814 tablet_replica.cc:330] T f697174a08a24bf6b4a77f69ed0062f6 P 28fa0e82e3e0496c89d30fdb80df848f: stopping tablet replica
I20260928 02:18:23.210315 12814 raft_consensus.cc:2229] T f697174a08a24bf6b4a77f69ed0062f6 P 28fa0e82e3e0496c89d30fdb80df848f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:18:23.210515 12814 raft_consensus.cc:2258] T f697174a08a24bf6b4a77f69ed0062f6 P 28fa0e82e3e0496c89d30fdb80df848f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:23.211464 12649 tablet_service.cc:1514] Processing DeleteTablet for tablet f697174a08a24bf6b4a77f69ed0062f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:23 UTC) from {username='oryx', principal='oryx/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:43898
I20260928 02:18:23.212066 12816 tablet_replica.cc:330] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8: stopping tablet replica
I20260928 02:18:23.212235 12816 raft_consensus.cc:2229] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:18:23.212251 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 12305
I20260928 02:18:23.212284 12814 ts_tablet_manager.cc:1905] T f697174a08a24bf6b4a77f69ed0062f6 P 28fa0e82e3e0496c89d30fdb80df848f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:23.212579 12816 raft_consensus.cc:2258] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:23.213678 12816 ts_tablet_manager.cc:1905] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:23.219431 12814 ts_tablet_manager.cc:1918] T f697174a08a24bf6b4a77f69ed0062f6 P 28fa0e82e3e0496c89d30fdb80df848f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:18:23.219619 12814 log.cc:1196] T f697174a08a24bf6b4a77f69ed0062f6 P 28fa0e82e3e0496c89d30fdb80df848f: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/f697174a08a24bf6b4a77f69ed0062f6
I20260928 02:18:23.220100 12814 ts_tablet_manager.cc:1939] T f697174a08a24bf6b4a77f69ed0062f6 P 28fa0e82e3e0496c89d30fdb80df848f: Deleting consensus metadata
I20260928 02:18:23.220300 12815 ts_tablet_manager.cc:1905] T f697174a08a24bf6b4a77f69ed0062f6 P 20f2f1dffc9a478cb5182a25c66a1a9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:23.221246 12221 catalog_manager.cc:4841] TS 28fa0e82e3e0496c89d30fdb80df848f (127.4.140.193:41081): tablet f697174a08a24bf6b4a77f69ed0062f6 (table test-table [id=e38948a3b619478da6df96811c39606f]) successfully deleted
I20260928 02:18:23.226055 12815 ts_tablet_manager.cc:1918] T f697174a08a24bf6b4a77f69ed0062f6 P 20f2f1dffc9a478cb5182a25c66a1a9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:18:23.226209 12815 log.cc:1196] T f697174a08a24bf6b4a77f69ed0062f6 P 20f2f1dffc9a478cb5182a25c66a1a9a: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/f697174a08a24bf6b4a77f69ed0062f6
I20260928 02:18:23.226717 12815 ts_tablet_manager.cc:1939] T f697174a08a24bf6b4a77f69ed0062f6 P 20f2f1dffc9a478cb5182a25c66a1a9a: Deleting consensus metadata
I20260928 02:18:23.228048 12221 catalog_manager.cc:4841] TS 20f2f1dffc9a478cb5182a25c66a1a9a (127.4.140.194:45669): tablet f697174a08a24bf6b4a77f69ed0062f6 (table test-table [id=e38948a3b619478da6df96811c39606f]) successfully deleted
I20260928 02:18:23.229362 12816 ts_tablet_manager.cc:1918] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:18:23.230461 12816 log.cc:1196] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/f697174a08a24bf6b4a77f69ed0062f6
I20260928 02:18:23.230969 12816 ts_tablet_manager.cc:1939] T f697174a08a24bf6b4a77f69ed0062f6 P 86f236333f4148b886a2e7a81af6dff8: Deleting consensus metadata
I20260928 02:18:23.232195 12218 catalog_manager.cc:4841] TS 86f236333f4148b886a2e7a81af6dff8 (127.4.140.195:32857): tablet f697174a08a24bf6b4a77f69ed0062f6 (table test-table [id=e38948a3b619478da6df96811c39606f]) successfully deleted
Sep 28 02:18:23 dist-test-slave-h37r krb5kdc[12189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.140.193@KRBTEST.COM
I20260928 02:18:23.247606 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:23.214332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43934 (local address 127.4.140.193:41081)
0928 02:18:23.214519 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:23.214526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:23.230283 (+ 15757us) server_negotiation.cc:407] Connection header received
0928 02:18:23.230463 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:23.230468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:23.230558 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:23.230666 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:23.233477 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:23.234629 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:23.235843 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:23.236519 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:23.236654 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:23.240540 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:23.240561 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:23.240583 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:23.240627 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:23.244394 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:23.245095 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:23.245106 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:23.245109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:23.245288 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:23.245788 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:23.245793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:23.245797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:23.246311 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:23.246636 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:23.246924 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 02:18:23.247183 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:18:23.248409 12436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:23.371564 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 12305
I20260928 02:18:23.397449 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 12445
Sep 28 02:18:23 dist-test-slave-h37r krb5kdc[12189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.140.194@KRBTEST.COM
I20260928 02:18:23.412387 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:23.398145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55830 (local address 127.4.140.194:45669)
0928 02:18:23.398595 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:23.398600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:23.398615 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:18:23.398666 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:23.398669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:23.398729 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:23.398840 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:23.400237 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:23.400906 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:23.402157 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:23.402553 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:23.402694 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:23.406647 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:23.406660 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:23.406676 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:23.406710 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:23.409656 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:23.410287 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:23.410294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:23.410296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:23.410417 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:23.410758 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:23.410763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:23.410767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:23.411098 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:23.411372 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:23.411706 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 02:18:23.412073 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":195,"threads_started":1}
I20260928 02:18:23.413074 12575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:23.536537 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 12445
I20260928 02:18:23.558260 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 12584
Sep 28 02:18:23 dist-test-slave-h37r krb5kdc[12189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.140.195@KRBTEST.COM
I20260928 02:18:23.572971 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:23.559025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43918 (local address 127.4.140.195:32857)
0928 02:18:23.559219 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:23.559227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:23.559301 (+ 74us) server_negotiation.cc:407] Connection header received
0928 02:18:23.559567 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:23.559576 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:23.559640 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:23.559756 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:23.560965 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:23.561683 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:23.562972 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:23.563375 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:23.563508 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:23.567624 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:23.567639 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:23.567656 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:23.567690 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:23.570316 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:23.570959 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:23.570966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:23.570968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:23.571100 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:23.571452 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:23.571460 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:23.571465 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:23.571790 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:23.572271 (+ 481us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:23.572425 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 02:18:23.572650 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:18:23.573688 12714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:23.701210 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 12584
I20260928 02:18:23.724126 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 12202
I20260928 02:18:23.725337 12274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:23.839569 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 12202
2026-09-28T02:18:23Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7148 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:18:23 dist-test-slave-h37r krb5kdc[12865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:18:23 dist-test-slave-h37r krb5kdc[12865](info): set up 2 sockets
Sep 28 02:18:23 dist-test-slave-h37r krb5kdc[12865](info): commencing operation
krb5kdc: starting...
W20260928 02:18:24.948855 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:18:25 dist-test-slave-h37r krb5kdc[12865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:18:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:18:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:18:25.172097 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:34273
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:34273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:25.487483 12878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:25.487818 12878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:25.488000 12878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:25.495290 12878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:18:25.495433 12878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:25.495546 12878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:25.495641 12878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:18:25.495723 12878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:18:25.505815 12878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:34273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:34273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:25.507108 12878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:25.508406 12878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:25.520310 12884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:25.520403 12883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:25.520375 12886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:25.521143 12878 server_base.cc:1034] running on GCE node
I20260928 02:18:25.521979 12878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:25.523463 12878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:25.524731 12878 hybrid_clock.cc:648] HybridClock initialized: now 1790561905524677 us; error 56 us; skew 500 ppm
I20260928 02:18:25.525125 12878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T02:18:25Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1748 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:18:25 dist-test-slave-h37r krb5kdc[12892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:18:25 dist-test-slave-h37r krb5kdc[12892](info): set up 2 sockets
Sep 28 02:18:25 dist-test-slave-h37r krb5kdc[12892](info): commencing operation
krb5kdc: starting...
W20260928 02:18:26.680840 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:18:26 dist-test-slave-h37r krb5kdc[12892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:18:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:18:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:18:26.907685 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:36541
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:41827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:36541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:27.237123 12905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:27.237406 12905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:27.237566 12905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:27.244693 12905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:18:27.244792 12905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:27.244913 12905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:27.245021 12905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:18:27.245101 12905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:18:27.254794 12905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:41827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:36541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:36541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:27.256188 12905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:27.257522 12905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:27.268854 12913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:27.269055 12911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:27.269301 12910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:27.269842 12905 server_base.cc:1034] running on GCE node
I20260928 02:18:27.270582 12905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:27.271981 12905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:27.273216 12905 hybrid_clock.cc:648] HybridClock initialized: now 1790561907273169 us; error 47 us; skew 500 ppm
I20260928 02:18:27.273563 12905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:27 dist-test-slave-h37r krb5kdc[12892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561907, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:27.282188 12905 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:18:27.283691 12905 webserver.cc:460] Webserver started at http://127.4.140.254:36297/ using document root <none> and password file <none>
I20260928 02:18:27.284442 12905 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:27.284587 12905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:27.284895 12905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:27.287808 12905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "a012f17ff843417d8f46ddd31bde5d47"
format_stamp: "Formatted at 2026-09-28 02:18:27 on dist-test-slave-h37r"
server_key: "dfe817ea47434b53632bc948ad64e561"
server_key_iv: "d07a054ef1e453e977de87e0e1874035"
server_key_version: "encryptionkey@0"
I20260928 02:18:27.288694 12905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "a012f17ff843417d8f46ddd31bde5d47"
format_stamp: "Formatted at 2026-09-28 02:18:27 on dist-test-slave-h37r"
server_key: "dfe817ea47434b53632bc948ad64e561"
server_key_iv: "d07a054ef1e453e977de87e0e1874035"
server_key_version: "encryptionkey@0"
I20260928 02:18:27.293650 12905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:18:27.297065 12920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:27.298555 12905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:18:27.298764 12905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "a012f17ff843417d8f46ddd31bde5d47"
format_stamp: "Formatted at 2026-09-28 02:18:27 on dist-test-slave-h37r"
server_key: "dfe817ea47434b53632bc948ad64e561"
server_key_iv: "d07a054ef1e453e977de87e0e1874035"
server_key_version: "encryptionkey@0"
I20260928 02:18:27.298964 12905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:27.325281 12905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:27.329360 12905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:27.329653 12905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:27.347908 12905 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:36541
I20260928 02:18:27.347997 12971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:36541 every 8 connection(s)
I20260928 02:18:27.349293 12905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:18:27.352989 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:27.359414 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 12905
I20260928 02:18:27.359683 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:18:27.360422 4659 external_mini_cluster.cc:1428] Setting key f5c23dc06d6961794901e362874ecf4b
I20260928 02:18:27.363109 12972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47: Bootstrap starting.
I20260928 02:18:27.367180 12972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:27.368255 12972 log.cc:824] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:27.371137 12972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47: No bootstrap required, opened a new log
Sep 28 02:18:27 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:18:27.380584 12972 raft_consensus.cc:348] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012f17ff843417d8f46ddd31bde5d47" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36541 } }
I20260928 02:18:27.381013 12972 raft_consensus.cc:374] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:27.381122 12972 raft_consensus.cc:729] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a012f17ff843417d8f46ddd31bde5d47, State: Initialized, Role: FOLLOWER
I20260928 02:18:27.381806 12972 consensus_queue.cc:260] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012f17ff843417d8f46ddd31bde5d47" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36541 } }
I20260928 02:18:27.382097 12972 raft_consensus.cc:388] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:18:27.382241 12972 raft_consensus.cc:482] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:18:27.382414 12972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:27.385059 12975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:27.362522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.4.140.254:36541)
0928 02:18:27.363370 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:27.363388 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:27.363435 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:18:27.365916 (+ 2481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:27.365947 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:27.366444 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:27.366905 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:27.368370 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:27.370287 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:27.371569 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:27.372579 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:27.372747 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:27.376494 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:27.376505 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:27.376522 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:27.376563 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:27.380523 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:27.381210 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:27.381221 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:27.381229 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:27.381427 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:27.381833 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:27.381838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:27.381841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:27.382555 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:27.382838 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:27.383201 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 02:18:27.383647 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":614,"thread_start_us":351,"threads_started":1}
I20260928 02:18:27.385669 12972 raft_consensus.cc:504] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012f17ff843417d8f46ddd31bde5d47" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36541 } }
I20260928 02:18:27.386153 12972 leader_election.cc:302] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a012f17ff843417d8f46ddd31bde5d47; no voters:
I20260928 02:18:27.386814 12972 leader_election.cc:288] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:18:27.386940 12977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:18:27.388418 12977 raft_consensus.cc:686] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [term 1 LEADER]: Becoming Leader. State: Replica: a012f17ff843417d8f46ddd31bde5d47, State: Running, Role: LEADER
I20260928 02:18:27.388973 12977 consensus_queue.cc:237] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012f17ff843417d8f46ddd31bde5d47" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36541 } }
I20260928 02:18:27.389186 12972 sys_catalog.cc:564] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:18:27.391927 12978 sys_catalog.cc:455] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a012f17ff843417d8f46ddd31bde5d47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012f17ff843417d8f46ddd31bde5d47" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36541 } } }
I20260928 02:18:27.392303 12978 sys_catalog.cc:458] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:27.392213 12979 sys_catalog.cc:455] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a012f17ff843417d8f46ddd31bde5d47. Latest consensus state: current_term: 1 leader_uuid: "a012f17ff843417d8f46ddd31bde5d47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012f17ff843417d8f46ddd31bde5d47" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36541 } } }
I20260928 02:18:27.392596 12979 sys_catalog.cc:458] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:27.399039 12985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:18:27.404786 12985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:18:27.414129 12985 catalog_manager.cc:1347] Generated new cluster ID: 7c40ccf523cc4250b547137314cb93c3
I20260928 02:18:27.414263 12985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:18:27.423179 12985 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:18:27.424580 12985 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:18:27.433921 12985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47: Generated new TSK 0
I20260928 02:18:27.434777 12985 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
I20260928 02:18:27.499456 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:36541
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:41827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:27.831622 12998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:27.831959 12998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:27.832146 12998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:27.839509 12998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:27.839618 12998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:27.839800 12998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:18:27.849660 12998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:41827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:36541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:27.851257 12998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:27.852744 12998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:27.865990 13003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:27.866047 13004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:27.866089 13006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:27.866925 12998 server_base.cc:1034] running on GCE node
I20260928 02:18:27.867525 12998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:27.868553 12998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:27.869783 12998 hybrid_clock.cc:648] HybridClock initialized: now 1790561907869734 us; error 32 us; skew 500 ppm
I20260928 02:18:27.870102 12998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:27 dist-test-slave-h37r krb5kdc[12892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561907, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:27.873719 12998 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:18:27.875226 12998 webserver.cc:460] Webserver started at http://127.4.140.193:37761/ using document root <none> and password file <none>
I20260928 02:18:27.876029 12998 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:27.876139 12998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:27.876484 12998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:27.879446 12998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "d88cd88debb944a0a69c5963b8a6d955"
format_stamp: "Formatted at 2026-09-28 02:18:27 on dist-test-slave-h37r"
server_key: "7bfad84511817829bcfd0624fb942c8d"
server_key_iv: "0230d42ebe37bb493aadaa289e2f8c16"
server_key_version: "encryptionkey@0"
I20260928 02:18:27.880322 12998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "d88cd88debb944a0a69c5963b8a6d955"
format_stamp: "Formatted at 2026-09-28 02:18:27 on dist-test-slave-h37r"
server_key: "7bfad84511817829bcfd0624fb942c8d"
server_key_iv: "0230d42ebe37bb493aadaa289e2f8c16"
server_key_version: "encryptionkey@0"
I20260928 02:18:27.885813 12998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:18:27.889484 13013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:27.890991 12998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 02:18:27.891173 12998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "d88cd88debb944a0a69c5963b8a6d955"
format_stamp: "Formatted at 2026-09-28 02:18:27 on dist-test-slave-h37r"
server_key: "7bfad84511817829bcfd0624fb942c8d"
server_key_iv: "0230d42ebe37bb493aadaa289e2f8c16"
server_key_version: "encryptionkey@0"
I20260928 02:18:27.891356 12998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:27.916369 12998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:27.920457 12998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:27.920743 12998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:27.921731 12998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:27.923036 12998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:27.923111 12998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:27.923233 12998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:27.923295 12998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:27.958979 12998 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:41269
I20260928 02:18:27.959014 13125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:41269 every 8 connection(s)
I20260928 02:18:27.960438 12998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:18:27.963726 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 12998
I20260928 02:18:27.964012 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:18:27.964886 4659 external_mini_cluster.cc:1428] Setting key 51d0f26f3bab520396d72c0ed1be06a7
Sep 28 02:18:27 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561907, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:18:27.983197 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:27.963174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:59147 (local address 127.4.140.254:36541)
0928 02:18:27.963641 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:27.963646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:27.964873 (+ 1227us) server_negotiation.cc:407] Connection header received
0928 02:18:27.967251 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:27.967257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:27.967343 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:27.967536 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:27.969482 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:27.970781 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:27.972448 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:27.973260 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:27.973432 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:27.977469 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:27.977474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:27.977478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:27.977510 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:27.980169 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:27.980910 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:27.980920 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:27.980923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:27.981040 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:27.981466 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:27.981470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:27.981473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:27.981732 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:27.981886 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:27.982603 (+ 717us) server_negotiation.cc:299] Negotiation successful
0928 02:18:27.982818 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":255,"threads_started":1}
I20260928 02:18:27.984278 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:27.963915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.193:59147)
0928 02:18:27.964712 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:27.964750 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:27.966905 (+ 2155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:27.967665 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:27.967673 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:27.968438 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:27.969261 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:27.969286 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:27.970927 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:27.970934 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:27.972301 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:27.972313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:27.973486 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:27.973490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:27.973691 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:27.974425 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:27.974448 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:27.977217 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:27.980313 (+ 3096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:27.980322 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:27.980341 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:27.980778 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:27.981153 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:27.981158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:27.981161 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:27.981347 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:27.981860 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:27.981869 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:27.982222 (+ 353us) client_negotiation.cc:769] Sending connection context
0928 02:18:27.982499 (+ 277us) client_negotiation.cc:240] Negotiation successful
0928 02:18:27.983072 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":316,"threads_started":1}
I20260928 02:18:27.985942 13126 heartbeater.cc:347] Connected to a master server at 127.4.140.254:36541
I20260928 02:18:27.986327 13126 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:27.987282 13126 heartbeater.cc:511] Master 127.4.140.254:36541 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.140.194@KRBTEST.COM; defaulting to no policy
I20260928 02:18:27.989966 12937 ts_manager.cc:194] Registered new tserver with Master: d88cd88debb944a0a69c5963b8a6d955 (127.4.140.193:41269)
I20260928 02:18:27.991806 12937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:59147
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
I20260928 02:18:28.018784 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:36541
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:41827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:28.345057 13132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:28.345332 13132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:28.345530 13132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:28.352838 13132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:28.352933 13132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:28.353154 13132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:18:28.362730 13132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:41827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:36541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:28.364006 13132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:28.365310 13132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:28.378523 13137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:28.378562 13140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:28.378626 13138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:28.379189 13132 server_base.cc:1034] running on GCE node
I20260928 02:18:28.379796 13132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:28.380743 13132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:28.381971 13132 hybrid_clock.cc:648] HybridClock initialized: now 1790561908381920 us; error 59 us; skew 500 ppm
I20260928 02:18:28.382295 13132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:28 dist-test-slave-h37r krb5kdc[12892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561908, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:28.385510 13132 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:18:28.386907 13132 webserver.cc:460] Webserver started at http://127.4.140.194:37503/ using document root <none> and password file <none>
I20260928 02:18:28.387575 13132 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:28.387718 13132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:28.388077 13132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:28.391273 13132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "6ba64b68d295492683ca6dc6738e4b23"
format_stamp: "Formatted at 2026-09-28 02:18:28 on dist-test-slave-h37r"
server_key: "f1ad5fec251fe2f3a74fb017ccb4fbf3"
server_key_iv: "1e44c0f05881d1329e4fad2a56314ccb"
server_key_version: "encryptionkey@0"
I20260928 02:18:28.392105 13132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "6ba64b68d295492683ca6dc6738e4b23"
format_stamp: "Formatted at 2026-09-28 02:18:28 on dist-test-slave-h37r"
server_key: "f1ad5fec251fe2f3a74fb017ccb4fbf3"
server_key_iv: "1e44c0f05881d1329e4fad2a56314ccb"
server_key_version: "encryptionkey@0"
I20260928 02:18:28.397351 13132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 02:18:28.400842 13147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:28.402213 13132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 02:18:28.402386 13132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "6ba64b68d295492683ca6dc6738e4b23"
format_stamp: "Formatted at 2026-09-28 02:18:28 on dist-test-slave-h37r"
server_key: "f1ad5fec251fe2f3a74fb017ccb4fbf3"
server_key_iv: "1e44c0f05881d1329e4fad2a56314ccb"
server_key_version: "encryptionkey@0"
I20260928 02:18:28.402619 13132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:28.415804 13132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:28.420243 13132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:28.420542 13132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:28.421515 13132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:28.422816 13132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:28.422919 13132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:28.423066 13132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:28.423146 13132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:28.456447 13132 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:36167
I20260928 02:18:28.456488 13259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:36167 every 8 connection(s)
I20260928 02:18:28.457746 13132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:18:28.460333 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 13132
I20260928 02:18:28.460532 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:18:28.461053 4659 external_mini_cluster.cc:1428] Setting key db8775c60f35c8d98d659a3de69ed1d9
Sep 28 02:18:28 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561908, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:18:28.479780 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:28.460368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:50625 (local address 127.4.140.254:36541)
0928 02:18:28.460563 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:28.460567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:28.462325 (+ 1758us) server_negotiation.cc:407] Connection header received
0928 02:18:28.464690 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:28.464699 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:28.464785 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:28.464896 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:28.466402 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:28.467515 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:28.469127 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:28.469841 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:28.469937 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:28.474020 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:28.474024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:28.474027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:28.474057 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:28.476621 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:28.477335 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:28.477341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:28.477344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:28.477457 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:28.477877 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:28.477882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:28.477885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:28.478156 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:28.478306 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:28.479062 (+ 756us) server_negotiation.cc:299] Negotiation successful
0928 02:18:28.479319 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 02:18:28.480476 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:28.461086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.194:50625)
0928 02:18:28.462173 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:28.462220 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:28.464419 (+ 2199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:28.465097 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:28.465107 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:28.465657 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:28.466230 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:28.466249 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:28.467666 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:28.467670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:28.468977 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:28.468989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:28.470069 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:28.470078 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:28.470413 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:28.471162 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:28.471196 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:28.473797 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:28.476763 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:28.476772 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:28.476789 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:28.477206 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:28.477564 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:28.477569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:28.477574 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:28.477761 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:28.478279 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:28.478287 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:28.478621 (+ 334us) client_negotiation.cc:769] Sending connection context
0928 02:18:28.478891 (+ 270us) client_negotiation.cc:240] Negotiation successful
0928 02:18:28.479321 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":567,"threads_started":1}
I20260928 02:18:28.482018 13260 heartbeater.cc:347] Connected to a master server at 127.4.140.254:36541
I20260928 02:18:28.482352 13260 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.140.195@KRBTEST.COM; defaulting to no policy
I20260928 02:18:28.483103 13260 heartbeater.cc:511] Master 127.4.140.254:36541 requested a full tablet report, sending...
I20260928 02:18:28.484529 12937 ts_manager.cc:194] Registered new tserver with Master: 6ba64b68d295492683ca6dc6738e4b23 (127.4.140.194:36167)
I20260928 02:18:28.485507 12937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:50625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
I20260928 02:18:28.511518 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:36541
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:41827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:28.835500 13265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:28.835789 13265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:28.835973 13265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:28.843209 13265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:28.843323 13265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:28.843557 13265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:18:28.853459 13265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:41827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:36541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:28.854794 13265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:28.856178 13265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:28.869438 13270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:28.869513 13273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:28.869536 13271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:28.873512 13265 server_base.cc:1034] running on GCE node
I20260928 02:18:28.874208 13265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:28.875222 13265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:28.876451 13265 hybrid_clock.cc:648] HybridClock initialized: now 1790561908876404 us; error 43 us; skew 500 ppm
I20260928 02:18:28.876785 13265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:28 dist-test-slave-h37r krb5kdc[12892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561908, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:28.880075 13265 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:18:28.881733 13265 webserver.cc:460] Webserver started at http://127.4.140.195:44793/ using document root <none> and password file <none>
I20260928 02:18:28.882489 13265 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:28.882643 13265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:28.882961 13265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:28.886123 13265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "48bcac42616b4f25b9a88a4e06ac0fb8"
format_stamp: "Formatted at 2026-09-28 02:18:28 on dist-test-slave-h37r"
server_key: "e1c1b01aaf679433eb8336bfc2d59383"
server_key_iv: "3b81e13b68f1580b7ff65f7473ec0fb7"
server_key_version: "encryptionkey@0"
I20260928 02:18:28.886924 13265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "48bcac42616b4f25b9a88a4e06ac0fb8"
format_stamp: "Formatted at 2026-09-28 02:18:28 on dist-test-slave-h37r"
server_key: "e1c1b01aaf679433eb8336bfc2d59383"
server_key_iv: "3b81e13b68f1580b7ff65f7473ec0fb7"
server_key_version: "encryptionkey@0"
I20260928 02:18:28.892454 13265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:18:28.896382 13280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:28.897909 13265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:18:28.898072 13265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "48bcac42616b4f25b9a88a4e06ac0fb8"
format_stamp: "Formatted at 2026-09-28 02:18:28 on dist-test-slave-h37r"
server_key: "e1c1b01aaf679433eb8336bfc2d59383"
server_key_iv: "3b81e13b68f1580b7ff65f7473ec0fb7"
server_key_version: "encryptionkey@0"
I20260928 02:18:28.898290 13265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:28.943984 13265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:28.948189 13265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:28.948486 13265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:28.949460 13265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:28.950800 13265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:28.950878 13265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:28.951049 13265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:28.951100 13265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:28.984534 13265 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:36011
I20260928 02:18:28.984634 13392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:36011 every 8 connection(s)
I20260928 02:18:28.985895 13265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:18:28.993098 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 13265
I20260928 02:18:28.993304 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:18:28.993726 4659 external_mini_cluster.cc:1428] Setting key cbeb9a30854dbe19c1a91c95e8ffb9a9
I20260928 02:18:28.995016 13126 heartbeater.cc:503] Master 127.4.140.254:36541 was elected leader, sending a full tablet report...
Sep 28 02:18:28 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561908, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:18:29.007041 13395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:28.988529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:56535 (local address 127.4.140.254:36541)
0928 02:18:28.988976 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:28.988980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:28.989975 (+ 995us) server_negotiation.cc:407] Connection header received
0928 02:18:28.991885 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:28.991890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:28.992010 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:28.992133 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:28.993766 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:28.994858 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:28.997219 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:28.997622 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:28.997758 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:29.001270 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:29.001275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:29.001278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:29.001307 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:29.004090 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:29.004776 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:29.004783 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:29.004786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:29.004908 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:29.005331 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:29.005338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:29.005341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:29.005599 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:29.005759 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:29.006453 (+ 694us) server_negotiation.cc:299] Negotiation successful
0928 02:18:29.006667 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":250,"threads_started":1}
I20260928 02:18:29.008172 13396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:28.989114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.195:56535)
0928 02:18:28.989809 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:28.989852 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:28.991592 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:28.992380 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:28.992388 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:28.992906 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:28.993593 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:28.993614 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:28.995032 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:28.995035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:28.997051 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:28.997069 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:28.997846 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:28.997853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:28.998112 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:28.998809 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:28.998882 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:29.001047 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:29.004223 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:29.004232 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:29.004249 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:29.004655 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:29.005017 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:29.005022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:29.005026 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:29.005221 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:29.005735 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:29.005743 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:29.006087 (+ 344us) client_negotiation.cc:769] Sending connection context
0928 02:18:29.006377 (+ 290us) client_negotiation.cc:240] Negotiation successful
0928 02:18:29.006922 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":254,"threads_started":1}
I20260928 02:18:29.009677 13393 heartbeater.cc:347] Connected to a master server at 127.4.140.254:36541
I20260928 02:18:29.010027 13393 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:29.010761 13393 heartbeater.cc:511] Master 127.4.140.254:36541 requested a full tablet report, sending...
I20260928 02:18:29.012135 12936 ts_manager.cc:194] Registered new tserver with Master: 48bcac42616b4f25b9a88a4e06ac0fb8 (127.4.140.195:36011)
I20260928 02:18:29.013001 12936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:56535
I20260928 02:18:29.022200 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.193.keytab.
Entry for principal oryx/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.193.keytab.
WARNING: no policy specified for oryx/127.4.140.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.140.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.194.keytab.
Entry for principal oryx/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.194.keytab.
WARNING: no policy specified for oryx/127.4.140.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.140.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.195.keytab.
Entry for principal oryx/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.195.keytab.
I20260928 02:18:29.163285 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 12998
Sep 28 02:18:29 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:18:29.179467 13404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:29.164522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56142 (local address 127.4.140.193:41269)
0928 02:18:29.165505 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:29.165512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:29.165536 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:18:29.165600 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:29.165606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:29.165811 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:29.165945 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:29.167270 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:29.168320 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:29.169478 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:29.170029 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:29.170135 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:29.174093 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:29.174110 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:29.174133 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:29.174179 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:29.177089 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:29.177832 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:29.177839 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:29.177841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:29.177965 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:29.178254 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:29.178257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:29.178259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:29.178580 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:29.178796 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:29.178994 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 02:18:29.179189 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":689,"threads_started":1}
I20260928 02:18:29.180817 13121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:29.265201 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 12998
I20260928 02:18:29.281816 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 13132
Sep 28 02:18:29 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.194@KRBTEST.COM
I20260928 02:18:29.298117 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:29.282724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.4.140.194:36167)
0928 02:18:29.283286 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:29.283293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:29.283319 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:18:29.283376 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:29.283383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:29.283591 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:29.283724 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:29.285095 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:29.286103 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:29.287385 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:29.288009 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:29.288128 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:29.292718 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:29.292729 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:29.292744 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:29.292777 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:29.295600 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:29.296316 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:29.296323 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:29.296325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:29.296441 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:29.296766 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:29.296769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:29.296771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:29.297088 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:29.297319 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:29.297527 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 02:18:29.297770 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":293,"threads_started":1}
I20260928 02:18:29.299499 13255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:29.381166 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 13132
I20260928 02:18:29.397380 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 13265
Sep 28 02:18:29 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.195@KRBTEST.COM
I20260928 02:18:29.414130 13408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:29.398367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.4.140.195:36011)
0928 02:18:29.398950 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:29.398957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:29.398987 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:18:29.399049 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:29.399056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:29.399322 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:29.399454 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:29.400872 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:29.401909 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:29.403288 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:29.404077 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:29.404184 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:29.408506 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:29.408517 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:29.408531 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:29.408561 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:29.411518 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:29.412186 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:29.412192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:29.412194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:29.412310 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:29.412710 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:29.412713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:29.412715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:29.413039 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:29.413263 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:29.413543 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 02:18:29.413778 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":262,"threads_started":1}
I20260928 02:18:29.415390 13388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:29.503345 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 13265
I20260928 02:18:29.519433 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 12905
I20260928 02:18:29.520680 12967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:29.621981 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 12905
I20260928 02:18:29.642735 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:36541
--webserver_interface=127.4.140.254
--webserver_port=36297
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:41827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:36541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:29.971072 13411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:29.971369 13411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:29.971457 13411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:29.978797 13411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:18:29.978919 13411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:29.978965 13411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:29.979015 13411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:18:29.979059 13411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:18:29.989112 13411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:41827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:36541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:36541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=36297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:29.990446 13411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:29.991727 13411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:30.004544 13419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:30.004468 13417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:30.004521 13416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:30.005333 13411 server_base.cc:1034] running on GCE node
I20260928 02:18:30.006139 13411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:30.007625 13411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:30.008828 13411 hybrid_clock.cc:648] HybridClock initialized: now 1790561910008760 us; error 79 us; skew 500 ppm
I20260928 02:18:30.009227 13411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:30 dist-test-slave-h37r krb5kdc[12892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561910, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:30.012578 13411 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:18:30.014169 13411 webserver.cc:460] Webserver started at http://127.4.140.254:36297/ using document root <none> and password file <none>
I20260928 02:18:30.014842 13411 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:30.014953 13411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:30.020483 13411 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:18:30.023322 13426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:30.024835 13411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:18:30.025017 13411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "a012f17ff843417d8f46ddd31bde5d47"
format_stamp: "Formatted at 2026-09-28 02:18:27 on dist-test-slave-h37r"
server_key: "dfe817ea47434b53632bc948ad64e561"
server_key_iv: "d07a054ef1e453e977de87e0e1874035"
server_key_version: "encryptionkey@0"
I20260928 02:18:30.025972 13411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:30.075838 13411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:30.080506 13411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:30.080823 13411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:30.099290 13411 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:36541
I20260928 02:18:30.099298 13477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:36541 every 8 connection(s)
I20260928 02:18:30.100867 13411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:18:30.104413 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 13411
I20260928 02:18:30.105497 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:41269
--local_ip_for_outbound_sockets=127.4.140.193
--tserver_master_addrs=127.4.140.254:36541
--webserver_port=37761
--webserver_interface=127.4.140.193
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:41827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.140.193
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:18:30.108151 13478 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:18:30.111338 13478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47: Bootstrap starting.
I20260928 02:18:30.130092 13478 log.cc:824] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:30.139233 13478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:18:30.139812 13478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47: Bootstrap complete.
I20260928 02:18:30.148793 13478 raft_consensus.cc:348] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012f17ff843417d8f46ddd31bde5d47" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36541 } }
I20260928 02:18:30.149632 13478 raft_consensus.cc:729] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a012f17ff843417d8f46ddd31bde5d47, State: Initialized, Role: FOLLOWER
I20260928 02:18:30.150287 13478 consensus_queue.cc:260] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012f17ff843417d8f46ddd31bde5d47" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36541 } }
I20260928 02:18:30.150547 13478 raft_consensus.cc:388] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:18:30.150671 13478 raft_consensus.cc:482] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:18:30.150858 13478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:18:30.154534 13478 raft_consensus.cc:504] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012f17ff843417d8f46ddd31bde5d47" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36541 } }
I20260928 02:18:30.155011 13478 leader_election.cc:302] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a012f17ff843417d8f46ddd31bde5d47; no voters:
I20260928 02:18:30.155670 13478 leader_election.cc:288] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:18:30.155800 13482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:18:30.157080 13482 raft_consensus.cc:686] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [term 2 LEADER]: Becoming Leader. State: Replica: a012f17ff843417d8f46ddd31bde5d47, State: Running, Role: LEADER
I20260928 02:18:30.157612 13482 consensus_queue.cc:237] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012f17ff843417d8f46ddd31bde5d47" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36541 } }
I20260928 02:18:30.158020 13478 sys_catalog.cc:564] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:18:30.160064 13484 sys_catalog.cc:455] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a012f17ff843417d8f46ddd31bde5d47. Latest consensus state: current_term: 2 leader_uuid: "a012f17ff843417d8f46ddd31bde5d47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012f17ff843417d8f46ddd31bde5d47" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36541 } } }
I20260928 02:18:30.160503 13484 sys_catalog.cc:458] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:30.161103 13483 sys_catalog.cc:455] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a012f17ff843417d8f46ddd31bde5d47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012f17ff843417d8f46ddd31bde5d47" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36541 } } }
I20260928 02:18:30.161460 13483 sys_catalog.cc:458] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:30.163625 13488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:18:30.170512 13488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:18:30.172266 13488 catalog_manager.cc:1259] Loaded cluster ID: 7c40ccf523cc4250b547137314cb93c3
I20260928 02:18:30.172387 13488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:18:30.177429 13488 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:18:30.179224 13488 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a012f17ff843417d8f46ddd31bde5d47: Loaded TSK: 0
I20260928 02:18:30.181159 13488 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:18:30.467332 13480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:30.467617 13480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:30.467762 13480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:30.475106 13480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:30.475231 13480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:30.475426 13480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:18:30.485460 13480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:41827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.193.keytab
--principal=oryx/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:41269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=37761
--tserver_master_addrs=127.4.140.254:36541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:30.486887 13480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:30.488193 13480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:30.501374 13508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:30.501353 13506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:30.501724 13505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:30.502761 13480 server_base.cc:1034] running on GCE node
I20260928 02:18:30.503535 13480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:30.504817 13480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:30.506088 13480 hybrid_clock.cc:648] HybridClock initialized: now 1790561910506020 us; error 54 us; skew 500 ppm
I20260928 02:18:30.506462 13480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:30 dist-test-slave-h37r krb5kdc[12892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561910, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:30.510423 13480 init.cc:377] Logged in from keytab as oryx/127.4.140.193@KRBTEST.COM (short username oryx)
I20260928 02:18:30.512236 13480 webserver.cc:460] Webserver started at http://127.4.140.193:37761/ using document root <none> and password file <none>
I20260928 02:18:30.513149 13480 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:30.513294 13480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:30.520334 13480 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:18:30.523439 13515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:30.525122 13480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:18:30.525285 13480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "d88cd88debb944a0a69c5963b8a6d955"
format_stamp: "Formatted at 2026-09-28 02:18:27 on dist-test-slave-h37r"
server_key: "7bfad84511817829bcfd0624fb942c8d"
server_key_iv: "0230d42ebe37bb493aadaa289e2f8c16"
server_key_version: "encryptionkey@0"
I20260928 02:18:30.526585 13480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:30.551520 13480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:30.555719 13480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:30.556036 13480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:30.557112 13480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:30.558547 13480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:30.558681 13480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:30.558801 13480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:30.558879 13480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:30.591599 13480 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:41269
I20260928 02:18:30.591689 13627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:41269 every 8 connection(s)
I20260928 02:18:30.593204 13480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:18:30.598421 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 13480
I20260928 02:18:30.599071 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:36167
--local_ip_for_outbound_sockets=127.4.140.194
--tserver_master_addrs=127.4.140.254:36541
--webserver_port=37503
--webserver_interface=127.4.140.194
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:41827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.140.194
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:18:30 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:30 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:18:30.943357 13632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:30.943653 13632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:30.943804 13632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:30.951047 13632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:30.951166 13632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:30.951381 13632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:18:30.962232 13632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:41827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.194.keytab
--principal=oryx/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:36167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=37503
--tserver_master_addrs=127.4.140.254:36541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:30.963593 13632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:30.964998 13632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:30.978664 13641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:30.978880 13638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:30.979182 13639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:30.979816 13632 server_base.cc:1034] running on GCE node
I20260928 02:18:30.980504 13632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:30.981632 13632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:30.982848 13632 hybrid_clock.cc:648] HybridClock initialized: now 1790561910982790 us; error 51 us; skew 500 ppm
I20260928 02:18:30.983253 13632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:30 dist-test-slave-h37r krb5kdc[12892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561910, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:30.986618 13632 init.cc:377] Logged in from keytab as oryx/127.4.140.194@KRBTEST.COM (short username oryx)
I20260928 02:18:30.988319 13632 webserver.cc:460] Webserver started at http://127.4.140.194:37503/ using document root <none> and password file <none>
I20260928 02:18:30.989022 13632 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:30.989173 13632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:30.994809 13632 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260928 02:18:30.998000 13648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:30.999730 13632 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260928 02:18:30.999946 13632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "6ba64b68d295492683ca6dc6738e4b23"
format_stamp: "Formatted at 2026-09-28 02:18:28 on dist-test-slave-h37r"
server_key: "f1ad5fec251fe2f3a74fb017ccb4fbf3"
server_key_iv: "1e44c0f05881d1329e4fad2a56314ccb"
server_key_version: "encryptionkey@0"
I20260928 02:18:31.000936 13632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:31.015758 13632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:31.020151 13632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:31.020430 13632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:31.021498 13632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:31.023015 13632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:31.023195 13632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:31.023348 13632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:31.023437 13632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:31.067970 13632 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:36167
I20260928 02:18:31.068009 13760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:36167 every 8 connection(s)
I20260928 02:18:31.069496 13632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:18:31.076709 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 13632
I20260928 02:18:31.077349 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:36011
--local_ip_for_outbound_sockets=127.4.140.195
--tserver_master_addrs=127.4.140.254:36541
--webserver_port=44793
--webserver_interface=127.4.140.195
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:41827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.140.195
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:18:31 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:31 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:18:31.441202 13765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:31.441522 13765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:31.441699 13765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:31.449062 13765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:31.449463 13765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:31.449769 13765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:18:31.463125 13765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:41827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/krb5kdc/oryx_127.4.140.195.keytab
--principal=oryx/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:36011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=44793
--tserver_master_addrs=127.4.140.254:36541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:31.464715 13765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:31.466152 13765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:31.487562 13771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:31.487612 13772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:31.495800 13774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:31.496681 13765 server_base.cc:1034] running on GCE node
I20260928 02:18:31.498992 13765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:31.500947 13765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:31.502429 13765 hybrid_clock.cc:648] HybridClock initialized: now 1790561911502329 us; error 77 us; skew 500 ppm
I20260928 02:18:31.503067 13765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:31 dist-test-slave-h37r krb5kdc[12892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561911, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:31.510149 13765 init.cc:377] Logged in from keytab as oryx/127.4.140.195@KRBTEST.COM (short username oryx)
I20260928 02:18:31.513105 13765 webserver.cc:460] Webserver started at http://127.4.140.195:44793/ using document root <none> and password file <none>
I20260928 02:18:31.514559 13765 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:31.514776 13765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:31.526072 13765 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.008s sys 0.000s
I20260928 02:18:31.531635 13781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:31.534415 13765 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260928 02:18:31.534807 13765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "48bcac42616b4f25b9a88a4e06ac0fb8"
format_stamp: "Formatted at 2026-09-28 02:18:28 on dist-test-slave-h37r"
server_key: "e1c1b01aaf679433eb8336bfc2d59383"
server_key_iv: "3b81e13b68f1580b7ff65f7473ec0fb7"
server_key_version: "encryptionkey@0"
I20260928 02:18:31.537077 13765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:31.598569 13765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:31.607345 13765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:31.607817 13765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:31.609611 13765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:31.613668 13765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:31.613839 13765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:31.613997 13765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:31.614111 13765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:31.682943 13765 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:36011
I20260928 02:18:31.683079 13893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:36011 every 8 connection(s)
I20260928 02:18:31.685879 13765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:18:31.687930 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 13765
Sep 28 02:18:31 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:31 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:31.725674 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:31.689376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.4.140.254:36541)
0928 02:18:31.696986 (+ 7610us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:31.697010 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:31.697063 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:18:31.697161 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:31.697169 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:31.697290 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:31.697473 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:31.707313 (+ 9840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:31.709074 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:31.710909 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:31.711559 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:31.711682 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:31.715472 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:31.715483 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:31.715504 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:31.715563 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:31.719934 (+ 4371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:31.720847 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:31.720857 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:31.720866 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:31.721076 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:31.721529 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:31.721537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:31.721544 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:31.722175 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:31.723689 (+ 1514us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:31.724005 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 02:18:31.724501 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7414,"thread_start_us":7221,"threads_started":1}
W20260928 02:18:32.097493 13631 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.490s user 0.006s sys 0.012s
W20260928 02:18:32.099426 13630 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:30.596330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:59827 (local address 127.4.140.254:36541)
0928 02:18:30.597096 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:30.597111 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:30.597384 (+ 273us) server_negotiation.cc:407] Connection header received
0928 02:18:30.599601 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:30.599624 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:30.600133 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:30.600562 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:30.602049 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:30.604069 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:30.605723 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:30.606366 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:30.606493 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:32.099091 (+1492598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.193:59827: BlockingRecv error: recv got EOF from 127.4.140.193:59827 (error 108)
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":287,"threads_started":1}
I20260928 02:18:32.102314 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:30.596424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.193:59827)
0928 02:18:30.597216 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:30.597259 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:30.598988 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:30.600761 (+ 1773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:30.600769 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:30.601288 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:30.601837 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:30.601854 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:30.604238 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:30.604242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:30.605569 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:30.605587 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:30.606616 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:30.606625 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:30.606960 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:30.607668 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:30.607707 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:32.098798 (+1491091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":356,"threads_started":1}
W20260928 02:18:32.105381 13628 heartbeater.cc:644] Failed to heartbeat to 127.4.140.254:36541 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.140.254:36541: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Sep 28 02:18:32 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:32 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:32.134820 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:32.106574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.193:33113)
0928 02:18:32.108555 (+ 1981us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:32.108587 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:32.108785 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:32.109534 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:32.109543 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:32.110271 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:32.110602 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:32.110619 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.112944 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.112953 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:32.115335 (+ 2382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:32.115358 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.123040 (+ 7682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.123050 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:32.123357 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:32.127124 (+ 3767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:32.127163 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:32.134321 (+ 7158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1289,"thread_start_us":15922,"threads_started":1}
W20260928 02:18:32.135052 13630 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:32.106623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:33113 (local address 127.4.140.254:36541)
0928 02:18:32.106876 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:32.106886 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:32.108680 (+ 1794us) server_negotiation.cc:407] Connection header received
0928 02:18:32.108938 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:32.108946 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:32.109075 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:32.109242 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:32.110873 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.112642 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.116142 (+ 3500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.117138 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.117275 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:32.134685 (+ 17410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.193:33113: BlockingRecv error: recv got EOF from 127.4.140.193:33113 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 02:18:32 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:32 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:32.152434 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:32.136150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.193:34807)
0928 02:18:32.136692 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:32.136718 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:32.136891 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:32.137404 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:32.137412 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:32.138007 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:32.138324 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:32.138340 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.140367 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.140372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:32.142462 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:32.142496 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.143971 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.143981 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:32.144250 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:32.145234 (+ 984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:32.145269 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:32.151950 (+ 6681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382}
W20260928 02:18:32.153211 13628 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:18:32.155216 13630 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:32.136270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:34807 (local address 127.4.140.254:36541)
0928 02:18:32.136504 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:32.136514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:32.136831 (+ 317us) server_negotiation.cc:407] Connection header received
0928 02:18:32.137051 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:32.137059 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:32.137192 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:32.137330 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:32.138541 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.140185 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.142679 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.143703 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.143877 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:32.154793 (+ 10916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.193:34807: BlockingRecv error: recv got EOF from 127.4.140.193:34807 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 02:18:32.473668 13764 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.384s user 0.001s sys 0.017s
W20260928 02:18:32.474928 13763 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:31.072605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:50197 (local address 127.4.140.254:36541)
0928 02:18:31.073164 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:31.073172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:31.074396 (+ 1224us) server_negotiation.cc:407] Connection header received
0928 02:18:31.078363 (+ 3967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:31.078372 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:31.078472 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:31.078602 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:31.081724 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:31.082921 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:31.086563 (+ 3642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:31.087312 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:31.087472 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:32.474623 (+1387151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.194:50197: BlockingRecv error: recv got EOF from 127.4.140.194:50197 (error 108)
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":308,"threads_started":1}
I20260928 02:18:32.475538 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:31.073071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.194:50197)
0928 02:18:31.074160 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:31.074242 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:31.077890 (+ 3648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:31.079027 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:31.079046 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:31.080250 (+ 1204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:31.081456 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:31.081492 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:31.083157 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:31.083166 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:31.086327 (+ 3161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:31.086351 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:31.087634 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:31.087645 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:31.088134 (+ 489us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:31.089183 (+ 1049us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:31.089228 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:32.474461 (+1385233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":348,"threads_started":1}
W20260928 02:18:32.476354 13761 heartbeater.cc:644] Failed to heartbeat to 127.4.140.254:36541 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.140.254:36541: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Sep 28 02:18:32 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:32 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:32.494107 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:32.477010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.194:32971)
0928 02:18:32.477204 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:32.477223 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:32.477316 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:32.477764 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:32.477767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:32.478082 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:32.478241 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:32.478249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.480286 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.480290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:32.481250 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:32.481261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.482685 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.482690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:32.482805 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:32.490400 (+ 7595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:32.490417 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:32.493898 (+ 3481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":77}
W20260928 02:18:32.494655 13763 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:32.477139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:32971 (local address 127.4.140.254:36541)
0928 02:18:32.477381 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:32.477391 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:32.477418 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:18:32.477496 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:32.477503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:32.477608 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:32.477761 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:32.478389 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.480112 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.481406 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.482385 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.482625 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:32.494281 (+ 11656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.194:32971: BlockingRecv error: recv got EOF from 127.4.140.194:32971 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 28 02:18:32 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:32 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:18:32.504690 13630 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:32.494963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:46261 (local address 127.4.140.254:36541)
0928 02:18:32.495133 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:32.495139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:32.495255 (+ 116us) server_negotiation.cc:407] Connection header received
0928 02:18:32.495438 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:32.495443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:32.495514 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:32.495611 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:32.496380 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.497309 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.498823 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.499244 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.499377 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:32.504517 (+ 5140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.194:46261: BlockingRecv error: recv got EOF from 127.4.140.194:46261 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:18:32.504742 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:32.494820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.194:46261)
0928 02:18:32.495147 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:32.495165 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:32.495310 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:32.495641 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:32.495645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:32.496036 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:32.496234 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:32.496250 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.497451 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.497456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:32.498680 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:32.498692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.499390 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.499407 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:32.499592 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:32.500258 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:32.500274 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:32.504326 (+ 4052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230}
W20260928 02:18:32.505131 13761 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:18:32.851079 13898 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.132s user 0.004s sys 0.011s
W20260928 02:18:32.852586 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:31.703349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:35509 (local address 127.4.140.254:36541)
0928 02:18:31.705558 (+ 2209us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:31.705566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:31.707958 (+ 2392us) server_negotiation.cc:407] Connection header received
0928 02:18:31.710070 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:31.710075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:31.710162 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:31.710233 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:31.712780 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:31.714113 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:31.716712 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:31.717451 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:31.717568 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:32.852162 (+1134594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.195:35509: BlockingRecv error: recv got EOF from 127.4.140.195:35509 (error 108)
Metrics: {"server-negotiator.queue_time_us":2029,"thread_start_us":2095,"threads_started":1}
I20260928 02:18:32.853053 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:31.702523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.195:35509)
0928 02:18:31.706166 (+ 3643us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:31.706230 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:31.709750 (+ 3520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:31.710850 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:31.710866 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:31.711636 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:31.712569 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:31.712597 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:31.714324 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:31.714331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:31.716553 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:31.716572 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:31.717969 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:31.717978 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:31.718282 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:31.718974 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:31.719001 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:32.851845 (+1132844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3282,"thread_start_us":3109,"threads_started":1}
W20260928 02:18:32.853976 13894 heartbeater.cc:644] Failed to heartbeat to 127.4.140.254:36541 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.140.254:36541: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Sep 28 02:18:32 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:32 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:32.862794 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:32.854592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.195:34757)
0928 02:18:32.854768 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:32.854783 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:32.854910 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:32.855231 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:32.855235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:32.855609 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:32.855773 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:32.855781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.856795 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.856800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:32.857820 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:32.857832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.858543 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.858558 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:32.858710 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:32.859353 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:32.859369 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:32.862594 (+ 3225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260928 02:18:32.862998 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:32.854766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:34757 (local address 127.4.140.254:36541)
0928 02:18:32.854935 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:32.854943 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:32.854964 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:18:32.855015 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:32.855019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:32.855095 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:32.855204 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:32.855933 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.856656 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.857974 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.858366 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.858493 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:32.862757 (+ 4264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.195:34757: BlockingRecv error: recv got EOF from 127.4.140.195:34757 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 02:18:32 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:32 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:32.871520 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:32.863505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.195:53955)
0928 02:18:32.863655 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:32.863678 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:32.863748 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:32.864076 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:32.864081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:32.864389 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:32.864564 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:32.864576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.865752 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.865757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:32.866688 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:32.866699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.867387 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.867392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:32.867585 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:32.868160 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:32.868176 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:32.871240 (+ 3064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 02:18:32.871665 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:32.863557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:53955 (local address 127.4.140.254:36541)
0928 02:18:32.863773 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:32.863778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:32.863792 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:18:32.863830 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:32.863834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:32.863884 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:32.864022 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:32.864715 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.865608 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.866836 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:32.867244 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:32.867375 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:32.871422 (+ 4047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.195:53955: BlockingRecv error: recv got EOF from 127.4.140.195:53955 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
W20260928 02:18:32.871847 13894 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 02:18:33 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:33 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:33.162384 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:33.153774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.193:60933)
0928 02:18:33.154399 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:33.154414 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:33.154522 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:33.154808 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:33.154811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:33.155220 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:33.155359 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:33.155367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:33.156325 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:33.156328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:33.157248 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:33.157259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:33.157875 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:33.157878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:33.158000 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:33.158665 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:33.158693 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:33.162027 (+ 3334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":416,"threads_started":1}
W20260928 02:18:33.162437 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:33.153795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:60933 (local address 127.4.140.254:36541)
0928 02:18:33.154016 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:33.154026 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:33.154492 (+ 466us) server_negotiation.cc:407] Connection header received
0928 02:18:33.154601 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:33.154605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:33.154687 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:33.154767 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:33.155477 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:33.156199 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:33.157361 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:33.157730 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:33.157818 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:33.162235 (+ 4417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.193:60933: BlockingRecv error: recv got EOF from 127.4.140.193:60933 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 02:18:33 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:33 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:33.514675 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:33.505713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.194:33375)
0928 02:18:33.506277 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:33.506298 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:33.506409 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:33.506701 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:33.506704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:33.507109 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:33.507247 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:33.507255 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:33.508230 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:33.508234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:33.509205 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:33.509216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:33.509875 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:33.509880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:33.509992 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:33.510625 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:33.510642 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:33.514325 (+ 3683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":341,"threads_started":1}
W20260928 02:18:33.514753 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:33.505769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:33375 (local address 127.4.140.254:36541)
0928 02:18:33.505942 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:33.505946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:33.506373 (+ 427us) server_negotiation.cc:407] Connection header received
0928 02:18:33.506509 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:33.506512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:33.506573 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:33.506657 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:33.507369 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:33.508106 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:33.509329 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:33.509728 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:33.509814 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:33.514516 (+ 4702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.194:33375: BlockingRecv error: recv got EOF from 127.4.140.194:33375 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 02:18:33 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:33 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:33.882166 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:33.872649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.195:56737)
0928 02:18:33.873173 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:33.873189 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:33.873308 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:33.873609 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:33.873612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:33.874005 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:33.874141 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:33.874148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:33.875294 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:33.875308 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:33.876789 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:33.876803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:33.877572 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:33.877576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:33.877688 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:33.878272 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:33.878289 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:33.881931 (+ 3642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":291,"threads_started":1}
W20260928 02:18:33.882323 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:33.872730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:56737 (local address 127.4.140.254:36541)
0928 02:18:33.873005 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:33.873011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:33.873298 (+ 287us) server_negotiation.cc:407] Connection header received
0928 02:18:33.873416 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:33.873419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:33.873482 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:33.873565 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:33.874273 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:33.875131 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:33.876934 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:33.877412 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:33.877504 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:33.882110 (+ 4606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.195:56737: BlockingRecv error: recv got EOF from 127.4.140.195:56737 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 02:18:34 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:34 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:34.172063 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:34.163463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.193:54217)
0928 02:18:34.164010 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:34.164026 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:34.164176 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:34.164532 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:34.164537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:34.164954 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:34.165121 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:34.165131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:34.166120 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:34.166128 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:34.167045 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:34.167055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:34.167742 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:34.167748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:34.167947 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:34.168500 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:34.168517 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:34.171691 (+ 3174us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":306,"threads_started":1}
W20260928 02:18:34.172188 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:34.163500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:54217 (local address 127.4.140.254:36541)
0928 02:18:34.163695 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:34.163703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:34.164126 (+ 423us) server_negotiation.cc:407] Connection header received
0928 02:18:34.164307 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:34.164312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:34.164383 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:34.164485 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:34.165266 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:34.165985 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:34.167190 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:34.167591 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:34.167727 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:34.171926 (+ 4199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.193:54217: BlockingRecv error: recv got EOF from 127.4.140.193:54217 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 02:18:34 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:34 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:18:34.524848 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:34.516110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:36017 (local address 127.4.140.254:36541)
0928 02:18:34.516295 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:34.516299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:34.516642 (+ 343us) server_negotiation.cc:407] Connection header received
0928 02:18:34.516833 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:34.516837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:34.516895 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:34.517007 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:34.517822 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:34.518527 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:34.519755 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:34.520240 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:34.520368 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:34.524650 (+ 4282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.194:36017: BlockingRecv error: recv got EOF from 127.4.140.194:36017 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:18:34.524835 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:34.516010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.194:36017)
0928 02:18:34.516542 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:34.516565 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:34.516718 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:34.517047 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:34.517053 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:34.517485 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:34.517675 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:34.517685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:34.518675 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:34.518680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:34.519604 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:34.519625 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:34.520401 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:34.520408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:34.520595 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:34.521149 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:34.521166 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:34.524463 (+ 3297us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":306,"threads_started":1}
Sep 28 02:18:34 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:34 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:34.892825 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:34.883297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.195:47417)
0928 02:18:34.884047 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:34.884066 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:34.884226 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:34.884645 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:34.884650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:34.885219 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:34.885470 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:34.885485 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:34.886657 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:34.886661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:34.887549 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:34.887560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:34.888251 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:34.888256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:34.888377 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:34.888941 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:34.888959 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:34.892582 (+ 3623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":463,"threads_started":1}
W20260928 02:18:34.893117 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:34.883558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:47417 (local address 127.4.140.254:36541)
0928 02:18:34.883753 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:34.883757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:34.884183 (+ 426us) server_negotiation.cc:407] Connection header received
0928 02:18:34.884384 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:34.884389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:34.884500 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:34.884578 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:34.885662 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:34.886525 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:34.887675 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:34.888123 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:34.888219 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:34.892840 (+ 4621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.195:47417: BlockingRecv error: recv got EOF from 127.4.140.195:47417 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 02:18:35 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:35 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:35.181535 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:35.173105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.193:35801)
0928 02:18:35.173636 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:35.173652 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:35.173761 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:35.174055 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:35.174059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:35.174384 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:35.174545 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:35.174552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:35.175589 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:35.175597 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:35.176794 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:35.176806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:35.177539 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:35.177543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:35.177678 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:35.178248 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:35.178264 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:35.181308 (+ 3044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":277,"threads_started":1}
W20260928 02:18:35.181758 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:35.173147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:35801 (local address 127.4.140.254:36541)
0928 02:18:35.173317 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:35.173321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:35.173720 (+ 399us) server_negotiation.cc:407] Connection header received
0928 02:18:35.173870 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:35.173873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:35.173932 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:35.174017 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:35.174700 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:35.175441 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:35.176936 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:35.177416 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:35.177513 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:35.181502 (+ 3989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.193:35801: BlockingRecv error: recv got EOF from 127.4.140.193:35801 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 02:18:35 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:35 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:35.534791 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:35.525830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.194:47763)
0928 02:18:35.526610 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:35.526652 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:35.526759 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:35.527022 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:35.527025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:35.527351 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:35.527508 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:35.527517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:35.528562 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:35.528566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:35.529523 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:35.529535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:35.530272 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:35.530277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:35.530449 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:35.531033 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:35.531050 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:35.534563 (+ 3513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":449,"threads_started":1}
W20260928 02:18:35.534996 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:35.525913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:47763 (local address 127.4.140.254:36541)
0928 02:18:35.526154 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:35.526163 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:35.526708 (+ 545us) server_negotiation.cc:407] Connection header received
0928 02:18:35.526840 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:35.526843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:35.526903 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:35.526978 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:35.527645 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:35.528436 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:35.529675 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:35.530103 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:35.530262 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:35.534775 (+ 4513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.194:47763: BlockingRecv error: recv got EOF from 127.4.140.194:47763 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 02:18:35 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:35 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:35.903082 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:35.893760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.195:38301)
0928 02:18:35.894329 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:35.894345 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:35.894459 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:35.894841 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:35.894844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:35.895203 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:35.895374 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:35.895383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:35.896705 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:35.896710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:35.897862 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:35.897873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:35.898621 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:35.898626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:35.898743 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:35.899484 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:35.899501 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:35.902826 (+ 3325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":291,"threads_started":1}
W20260928 02:18:35.903236 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:35.893897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:38301 (local address 127.4.140.254:36541)
0928 02:18:35.894064 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:35.894069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:35.894422 (+ 353us) server_negotiation.cc:407] Connection header received
0928 02:18:35.894617 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:35.894621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:35.894699 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:35.894793 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:35.895524 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:35.896508 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:35.898021 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:35.898459 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:35.898556 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:35.903020 (+ 4464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.195:38301: BlockingRecv error: recv got EOF from 127.4.140.195:38301 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 02:18:36 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:36 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:36.191653 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:36.182689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.193:55689)
0928 02:18:36.183170 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:36.183187 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:36.183333 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:36.183701 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:36.183705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:36.184135 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:36.184288 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:36.184297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:36.185303 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:36.185318 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:36.186414 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:36.186429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:36.187145 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:36.187150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:36.187330 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:36.188029 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:36.188046 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:36.191401 (+ 3355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":261,"threads_started":1}
W20260928 02:18:36.191874 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:36.182799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:55689 (local address 127.4.140.254:36541)
0928 02:18:36.183015 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:36.183022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:36.183268 (+ 246us) server_negotiation.cc:407] Connection header received
0928 02:18:36.183449 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:36.183455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:36.183542 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:36.183667 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:36.184433 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:36.185175 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:36.186597 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:36.186998 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:36.187135 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:36.191603 (+ 4468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.193:55689: BlockingRecv error: recv got EOF from 127.4.140.193:55689 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 02:18:36 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:36 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:36.544898 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:36.535735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.194:33021)
0928 02:18:36.536278 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:36.536298 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:36.536457 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:36.536821 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:36.536826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:36.537219 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:36.537375 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:36.537384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:36.538394 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:36.538402 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:36.539481 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:36.539495 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:36.540508 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:36.540516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:36.540675 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:36.541348 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:36.541365 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:36.544648 (+ 3283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":295,"threads_started":1}
W20260928 02:18:36.545033 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:36.535821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:33021 (local address 127.4.140.254:36541)
0928 02:18:36.536068 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:36.536075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:36.536388 (+ 313us) server_negotiation.cc:407] Connection header received
0928 02:18:36.536585 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:36.536591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:36.536666 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:36.536772 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:36.537520 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:36.538249 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:36.539649 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:36.540353 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:36.540478 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:36.544828 (+ 4350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.194:33021: BlockingRecv error: recv got EOF from 127.4.140.194:33021 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 28 02:18:36 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:36 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:36.913133 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:36.904153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.195:40825)
0928 02:18:36.904630 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:36.904664 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:36.904828 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:36.905223 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:36.905227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:36.905634 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:36.905787 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:36.905795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:36.906841 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:36.906849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:36.907868 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:36.907879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:36.908594 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:36.908600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:36.908792 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:36.909366 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:36.909383 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:36.912759 (+ 3376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":272,"threads_started":1}
W20260928 02:18:36.913235 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:36.904208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:40825 (local address 127.4.140.254:36541)
0928 02:18:36.904445 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:36.904453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:36.904757 (+ 304us) server_negotiation.cc:407] Connection header received
0928 02:18:36.904953 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:36.904958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:36.905086 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:36.905183 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:36.905938 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:36.906686 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:36.908034 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:36.908445 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:36.908579 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:36.912973 (+ 4394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.195:40825: BlockingRecv error: recv got EOF from 127.4.140.195:40825 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 02:18:37 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:37 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:37.202229 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:37.192693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.193:52391)
0928 02:18:37.193161 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:37.193177 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:37.193313 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:37.193642 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:37.193646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:37.194026 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:37.194231 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:37.194244 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:37.195781 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:37.195786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:37.196821 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:37.196833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:37.197500 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:37.197505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:37.197682 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:37.198352 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:37.198370 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:37.201887 (+ 3517us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":257,"threads_started":1}
W20260928 02:18:37.202299 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:37.192777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:52391 (local address 127.4.140.254:36541)
0928 02:18:37.193034 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:37.193040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:37.193265 (+ 225us) server_negotiation.cc:407] Connection header received
0928 02:18:37.193420 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:37.193423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:37.193486 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:37.193571 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:37.194455 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:37.195661 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:37.196969 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:37.197352 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:37.197492 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:37.202091 (+ 4599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.193:52391: BlockingRecv error: recv got EOF from 127.4.140.193:52391 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 28 02:18:37 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:37 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:37.563385 13915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:37.545878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.194:39827)
0928 02:18:37.546419 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:37.546452 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:37.546618 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:37.546941 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:37.546945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:37.547514 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:37.547728 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:37.547741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:37.548888 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:37.548891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:37.549872 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:37.549885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:37.550567 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:37.550572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:37.550712 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:37.559115 (+ 8403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:37.559136 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:37.563161 (+ 4025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":262,"threads_started":1}
W20260928 02:18:37.563611 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:37.546115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:39827 (local address 127.4.140.254:36541)
0928 02:18:37.546334 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:37.546342 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:37.546523 (+ 181us) server_negotiation.cc:407] Connection header received
0928 02:18:37.546753 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:37.546757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:37.546818 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:37.546896 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:37.547882 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:37.548763 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:37.549992 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:37.550395 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:37.550485 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:37.563352 (+ 12867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.194:39827: BlockingRecv error: recv got EOF from 127.4.140.194:39827 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 02:18:37 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:37 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:37.923571 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:37.914258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.195:33755)
0928 02:18:37.914756 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:37.914781 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:37.914900 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:37.915241 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:37.915245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:37.915674 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:37.915821 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:37.915830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:37.916895 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:37.916899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:37.917859 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:37.917906 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:37.918807 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:37.918812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:37.918997 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:37.919724 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:37.919740 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:37.923152 (+ 3412us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":267,"threads_started":1}
W20260928 02:18:37.923609 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:37.914401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:33755 (local address 127.4.140.254:36541)
0928 02:18:37.914658 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:37.914665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:37.914869 (+ 204us) server_negotiation.cc:407] Connection header received
0928 02:18:37.915002 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:37.915006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:37.915067 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:37.915164 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:37.916050 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:37.916760 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:37.918049 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:37.918657 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:37.918774 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:37.923388 (+ 4614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.195:33755: BlockingRecv error: recv got EOF from 127.4.140.195:33755 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 28 02:18:38 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:38 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:38.211964 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:38.203317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.193:46387)
0928 02:18:38.203828 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:38.203845 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:38.203988 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:38.204271 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:38.204274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:38.204692 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:38.204890 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:38.204899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:38.205854 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:38.205857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:38.206746 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:38.206757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:38.207436 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:38.207440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:38.207556 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:38.208259 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:38.208276 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:38.211687 (+ 3411us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":277,"threads_started":1}
W20260928 02:18:38.212199 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:38.203494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:46387 (local address 127.4.140.254:36541)
0928 02:18:38.203739 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:38.203744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:38.203943 (+ 199us) server_negotiation.cc:407] Connection header received
0928 02:18:38.204089 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:38.204094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:38.204151 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:38.204228 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:38.205006 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:38.205745 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:38.206871 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:38.207311 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:38.207401 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:38.211887 (+ 4486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.193:46387: BlockingRecv error: recv got EOF from 127.4.140.193:46387 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 02:18:38 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:38 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:38.573361 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:38.564259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.194:38153)
0928 02:18:38.564813 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:38.564845 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:38.564953 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:38.565246 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:38.565249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:38.565636 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:38.565800 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:38.565807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:38.566763 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:38.566768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:38.567717 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:38.567729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:38.568501 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:38.568506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:38.568634 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:38.569252 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:38.569269 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:38.573016 (+ 3747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":344,"threads_started":1}
W20260928 02:18:38.573423 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:38.564359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:38153 (local address 127.4.140.254:36541)
0928 02:18:38.564527 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:38.564531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:38.564917 (+ 386us) server_negotiation.cc:407] Connection header received
0928 02:18:38.565049 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:38.565053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:38.565123 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:38.565204 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:38.565920 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:38.566608 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:38.567852 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:38.568345 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:38.568447 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:38.573210 (+ 4763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.194:38153: BlockingRecv error: recv got EOF from 127.4.140.194:38153 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 02:18:38 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:38 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:38.934435 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:38.924942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.195:45121)
0928 02:18:38.925670 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:38.925693 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:38.925811 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:38.926362 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:38.926367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:38.926857 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:38.927023 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:38.927033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:38.928181 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:38.928185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:38.929097 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:38.929107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:38.929929 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:38.929941 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:38.930071 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:38.930809 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:38.930826 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:38.934213 (+ 3387us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":422,"threads_started":1}
W20260928 02:18:38.934696 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:38.925024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:45121 (local address 127.4.140.254:36541)
0928 02:18:38.925981 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:38.925988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:38.926007 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:18:38.926094 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:38.926099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:38.926187 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:38.926303 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:38.927167 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:38.928052 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:38.929251 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:38.929754 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:38.929856 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:38.934413 (+ 4557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.195:45121: BlockingRecv error: recv got EOF from 127.4.140.195:45121 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 28 02:18:39 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:39 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:39.223767 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:39.212834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.193:45039)
0928 02:18:39.213551 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:39.213572 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:39.213761 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:39.214160 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:39.214166 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:39.214697 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:39.214928 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:39.214939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:39.216249 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:39.216259 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:39.217586 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:39.217601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:39.218599 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:39.218606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:39.218810 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:39.219532 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:39.219553 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:39.223494 (+ 3941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":326,"threads_started":1}
W20260928 02:18:39.224081 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:39.212984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:45039 (local address 127.4.140.254:36541)
0928 02:18:39.213330 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:39.213341 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:39.213678 (+ 337us) server_negotiation.cc:407] Connection header received
0928 02:18:39.213909 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:39.213916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:39.214000 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:39.214125 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:39.215172 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:39.216088 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:39.217752 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:39.218415 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:39.218585 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:39.223770 (+ 5185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.193:45039: BlockingRecv error: recv got EOF from 127.4.140.193:45039 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 28 02:18:39 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:39 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:39.583442 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:39.574586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.194:53057)
0928 02:18:39.575294 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:39.575314 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:39.575425 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:39.575723 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:39.575726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:39.576143 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:39.576318 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:39.576326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:39.577284 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:39.577287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:39.578228 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:39.578238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:39.578870 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:39.578873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:39.578984 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:39.579630 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:39.579648 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:39.583047 (+ 3399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":310,"threads_started":1}
W20260928 02:18:39.583560 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:39.574615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:53057 (local address 127.4.140.254:36541)
0928 02:18:39.574848 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:39.574858 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:39.575395 (+ 537us) server_negotiation.cc:407] Connection header received
0928 02:18:39.575522 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:39.575526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:39.575597 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:39.575678 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:39.576442 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:39.577170 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:39.578342 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:39.578748 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:39.578837 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:39.583293 (+ 4456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.194:53057: BlockingRecv error: recv got EOF from 127.4.140.194:53057 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 02:18:39 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:39 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:39.944612 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:39.935412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.195:45755)
0928 02:18:39.936026 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:39.936043 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:39.936160 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:39.936478 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:39.936483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:39.936956 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:39.937188 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:39.937198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:39.938380 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:39.938384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:39.939353 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:39.939365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:39.940051 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:39.940055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:39.940189 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:39.940842 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:39.940859 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:39.944343 (+ 3484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":262,"threads_started":1}
W20260928 02:18:39.944801 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:39.935442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:45755 (local address 127.4.140.254:36541)
0928 02:18:39.935694 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:39.935703 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:39.936122 (+ 419us) server_negotiation.cc:407] Connection header received
0928 02:18:39.936258 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:39.936261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:39.936326 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:39.936408 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:39.937325 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:39.938226 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:39.939495 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:39.939919 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:39.940010 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:39.944570 (+ 4560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.195:45755: BlockingRecv error: recv got EOF from 127.4.140.195:45755 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 02:18:40 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:40 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:40.234601 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:40.224992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.193:59091)
0928 02:18:40.225569 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:40.225584 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:40.225748 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:40.226071 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:40.226076 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:40.226516 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:40.226675 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:40.226683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:40.227729 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:40.227735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:40.229181 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:40.229194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:40.230126 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:40.230131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:40.230240 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:40.230955 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:40.230985 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:40.234367 (+ 3382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":317,"threads_started":1}
W20260928 02:18:40.234848 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:40.225230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:59091 (local address 127.4.140.254:36541)
0928 02:18:40.225471 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:40.225478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:40.225671 (+ 193us) server_negotiation.cc:407] Connection header received
0928 02:18:40.225871 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:40.225879 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:40.225943 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:40.226046 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:40.226811 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:40.227590 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:40.229347 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:40.229963 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:40.230058 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:40.234557 (+ 4499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.193:59091: BlockingRecv error: recv got EOF from 127.4.140.193:59091 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 02:18:40 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:40 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:40.593587 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:40.584445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.194:58489)
0928 02:18:40.584918 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:40.584938 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:40.585059 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:40.585393 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:40.585396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:40.585794 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:40.585955 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:40.585964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:40.587058 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:40.587061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:40.588093 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:40.588105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:40.589074 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:40.589079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:40.589212 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:40.589789 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:40.589805 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:40.593236 (+ 3431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":216,"threads_started":1}
W20260928 02:18:40.593822 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:40.584685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:58489 (local address 127.4.140.254:36541)
0928 02:18:40.584914 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:40.584918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:40.585018 (+ 100us) server_negotiation.cc:407] Connection header received
0928 02:18:40.585160 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:40.585163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:40.585221 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:40.585300 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:40.586084 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:40.586937 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:40.588235 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:40.588907 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:40.589037 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:40.593533 (+ 4496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.194:58489: BlockingRecv error: recv got EOF from 127.4.140.194:58489 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 28 02:18:40 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:40 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:40.954754 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:40.945603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.195:48523)
0928 02:18:40.946143 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:40.946158 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:40.946329 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:40.946702 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:40.946706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:40.947146 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:40.947304 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:40.947313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:40.948411 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:40.948416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:40.949511 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:40.949523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:40.950492 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:40.950496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:40.950619 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:40.951267 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:40.951284 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:40.954461 (+ 3177us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":277,"threads_started":1}
W20260928 02:18:40.954916 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:40.945670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:48523 (local address 127.4.140.254:36541)
0928 02:18:40.945936 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:40.945945 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:40.946250 (+ 305us) server_negotiation.cc:407] Connection header received
0928 02:18:40.946458 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:40.946465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:40.946560 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:40.946670 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:40.947448 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:40.948279 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:40.949656 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:40.950096 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:40.950234 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:40.954666 (+ 4432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.195:48523: BlockingRecv error: recv got EOF from 127.4.140.195:48523 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 28 02:18:41 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:41 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:41.244518 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:41.235483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.193:34903)
0928 02:18:41.236006 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:41.236022 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:41.236178 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:41.236503 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:41.236506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:41.236911 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:41.237057 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:41.237065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:41.238110 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:41.238114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:41.239081 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:41.239092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:41.239706 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:41.239710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:41.239833 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:41.240516 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:41.240534 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:41.244166 (+ 3632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":256,"threads_started":1}
W20260928 02:18:41.244599 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:41.235551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:34903 (local address 127.4.140.254:36541)
0928 02:18:41.235770 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:41.235779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:41.236151 (+ 372us) server_negotiation.cc:407] Connection header received
0928 02:18:41.236319 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:41.236325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:41.236398 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:41.236482 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:41.237186 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:41.237988 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:41.239202 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:41.239578 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:41.239665 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:41.244363 (+ 4698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.193:34903: BlockingRecv error: recv got EOF from 127.4.140.193:34903 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 02:18:41 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:41 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:41.604127 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:41.594574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.194:48241)
0928 02:18:41.595068 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:41.595092 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:41.595266 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:41.595655 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:41.595662 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:41.596153 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:41.596292 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:41.596300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:41.597285 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:41.597288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:41.598223 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:41.598233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:41.598872 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:41.598876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:41.599001 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:41.599714 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:41.599730 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:41.603737 (+ 4007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":283,"threads_started":1}
W20260928 02:18:41.604239 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:41.594680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:48241 (local address 127.4.140.254:36541)
0928 02:18:41.594860 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:41.594867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:41.595171 (+ 304us) server_negotiation.cc:407] Connection header received
0928 02:18:41.595400 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:41.595406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:41.595485 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:41.595594 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:41.596427 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:41.597174 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:41.598343 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:41.598743 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:41.598831 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:41.603940 (+ 5109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.194:48241: BlockingRecv error: recv got EOF from 127.4.140.194:48241 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 02:18:41 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:41 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:41.965329 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:41.955825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.195:44641)
0928 02:18:41.956360 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:41.956375 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:41.956546 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:41.956883 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:41.956888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:41.957321 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:41.957485 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:41.957498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:41.958889 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:41.958897 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:41.959946 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:41.959962 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:41.960683 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:41.960688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:41.960820 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:41.961391 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:41.961409 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:41.964987 (+ 3578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":308,"threads_started":1}
W20260928 02:18:41.965364 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:41.955948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:44641 (local address 127.4.140.254:36541)
0928 02:18:41.956212 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:41.956220 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:41.956464 (+ 244us) server_negotiation.cc:407] Connection header received
0928 02:18:41.956681 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:41.956691 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:41.956751 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:41.956854 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:41.957695 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:41.958693 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:41.960094 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:41.960514 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:41.960630 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:41.965168 (+ 4538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.195:44641: BlockingRecv error: recv got EOF from 127.4.140.195:44641 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 28 02:18:42 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:42 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:42.254453 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:42.245644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.193:36159)
0928 02:18:42.246147 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:42.246163 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:42.246277 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:42.246567 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:42.246571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:42.247005 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:42.247156 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:42.247164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:42.248145 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:42.248149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:42.249316 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:42.249332 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:42.250253 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:42.250258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:42.250369 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:42.250901 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:42.250917 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:42.254116 (+ 3199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":265,"threads_started":1}
W20260928 02:18:42.254561 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:42.245701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:36159 (local address 127.4.140.254:36541)
0928 02:18:42.245863 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:42.245867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:42.246236 (+ 369us) server_negotiation.cc:407] Connection header received
0928 02:18:42.246368 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:42.246371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:42.246462 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:42.246545 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:42.247280 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:42.248017 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:42.249478 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:42.250121 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:42.250214 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:42.254302 (+ 4088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.193:36159: BlockingRecv error: recv got EOF from 127.4.140.193:36159 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 02:18:42 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:42 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:42.620865 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:42.605176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.194:37435)
0928 02:18:42.605699 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:42.605719 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:42.605836 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:42.606147 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:42.606150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:42.606474 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:42.606694 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:42.606708 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:42.607947 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:42.607952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:42.609223 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:42.609235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:42.609879 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:42.609883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:42.609998 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:42.616753 (+ 6755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:42.616771 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:42.620510 (+ 3739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":295,"threads_started":1}
W20260928 02:18:42.621042 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:42.605214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:37435 (local address 127.4.140.254:36541)
0928 02:18:42.605395 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:42.605399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:42.605801 (+ 402us) server_negotiation.cc:407] Connection header received
0928 02:18:42.605949 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:42.605953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:42.606015 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:42.606117 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:42.606837 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:42.607781 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:42.609350 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:42.609752 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:42.609845 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:42.620728 (+ 10883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.194:37435: BlockingRecv error: recv got EOF from 127.4.140.194:37435 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 02:18:42 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:42 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:42.974977 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:42.966320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.195:43319)
0928 02:18:42.966833 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:42.966849 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:42.966964 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:42.967244 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:42.967247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:42.967587 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:42.967742 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:42.967750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:42.968690 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:42.968694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:42.969662 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:42.969673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:42.970336 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:42.970342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:42.970511 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:42.971192 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:42.971238 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:42.974621 (+ 3383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":307,"threads_started":1}
W20260928 02:18:42.975078 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:42.966526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:43319 (local address 127.4.140.254:36541)
0928 02:18:42.966806 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:42.966813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:42.966928 (+ 115us) server_negotiation.cc:407] Connection header received
0928 02:18:42.967060 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:42.967063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:42.967122 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:42.967202 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:42.967860 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:42.968576 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:42.969785 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:42.970191 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:42.970283 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:42.974825 (+ 4542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.195:43319: BlockingRecv error: recv got EOF from 127.4.140.195:43319 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
Sep 28 02:18:43 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:43 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:43.264760 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:43.255423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.193:51873)
0928 02:18:43.255976 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:43.255991 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:43.256155 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:43.256504 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:43.256510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:43.256940 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:43.257107 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:43.257116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:43.258220 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:43.258228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:43.259539 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:43.259556 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:43.260383 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:43.260389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:43.260563 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:43.261182 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:43.261200 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:43.264542 (+ 3342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":290,"threads_started":1}
W20260928 02:18:43.264930 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:43.255584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:51873 (local address 127.4.140.254:36541)
0928 02:18:43.255810 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:43.255817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:43.256077 (+ 260us) server_negotiation.cc:407] Connection header received
0928 02:18:43.256284 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:43.256296 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:43.256370 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:43.256476 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:43.257260 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:43.258088 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:43.259676 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:43.260212 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:43.260347 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:43.264723 (+ 4376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.193:51873: BlockingRecv error: recv got EOF from 127.4.140.193:51873 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 02:18:43 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:43 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:43.631018 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:43.622020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.194:43563)
0928 02:18:43.622478 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:43.622496 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:43.622640 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:43.622962 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:43.622967 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:43.623379 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:43.623541 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:43.623549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:43.624665 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:43.624677 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:43.625628 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:43.625639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:43.626333 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:43.626339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:43.626506 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:43.627117 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:43.627134 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:43.630646 (+ 3512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":248,"threads_started":1}
W20260928 02:18:43.631076 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:43.622086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:43563 (local address 127.4.140.254:36541)
0928 02:18:43.622311 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:43.622319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:43.622571 (+ 252us) server_negotiation.cc:407] Connection header received
0928 02:18:43.622761 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:43.622766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:43.622838 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:43.622930 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:43.623677 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:43.624523 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:43.625771 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:43.626187 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:43.626319 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:43.630827 (+ 4508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.194:43563: BlockingRecv error: recv got EOF from 127.4.140.194:43563 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 02:18:43 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:43 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:43.984727 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:43.975889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.195:41067)
0928 02:18:43.976376 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:43.976391 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:43.976500 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:43.976794 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:43.976797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:43.977171 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:43.977338 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:43.977345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:43.978293 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:43.978298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:43.979264 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:43.979275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:43.980005 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:43.980010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:43.980164 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:43.980735 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:43.980753 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:43.984378 (+ 3625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":246,"threads_started":1}
W20260928 02:18:43.984779 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:43.976002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:41067 (local address 127.4.140.254:36541)
0928 02:18:43.976232 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:43.976238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:43.976474 (+ 236us) server_negotiation.cc:407] Connection header received
0928 02:18:43.976594 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:43.976597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:43.976668 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:43.976748 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:43.977452 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:43.978151 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:43.979395 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:43.979785 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:43.979874 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:43.984553 (+ 4679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.195:41067: BlockingRecv error: recv got EOF from 127.4.140.195:41067 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 02:18:44 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:44 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:44.274960 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:44.265814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.193:50339)
0928 02:18:44.266348 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:44.266364 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:44.266474 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:44.266818 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:44.266822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:44.267194 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:44.267356 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:44.267364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:44.268406 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:44.268410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:44.269336 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:44.269347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:44.270035 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:44.270039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:44.270174 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:44.270842 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:44.270860 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:44.274628 (+ 3768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":319,"threads_started":1}
W20260928 02:18:44.275188 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:44.265969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:50339 (local address 127.4.140.254:36541)
0928 02:18:44.266162 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:44.266167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:44.266469 (+ 302us) server_negotiation.cc:407] Connection header received
0928 02:18:44.266581 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:44.266586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:44.266668 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:44.266773 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:44.267507 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:44.268283 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:44.269461 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:44.269916 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:44.270008 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:44.274901 (+ 4893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.193:50339: BlockingRecv error: recv got EOF from 127.4.140.193:50339 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 02:18:44 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:44 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:44.640908 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:44.632092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.194:47151)
0928 02:18:44.632557 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:44.632576 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:44.632698 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:44.633032 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:44.633036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:44.633430 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:44.633587 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:44.633595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:44.634591 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:44.634595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:44.635533 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:44.635544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:44.636242 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:44.636247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:44.636390 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:44.637077 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:44.637095 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:44.640611 (+ 3516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":280,"threads_started":1}
W20260928 02:18:44.641156 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:44.632115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:47151 (local address 127.4.140.254:36541)
0928 02:18:44.632327 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:44.632337 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:44.632662 (+ 325us) server_negotiation.cc:407] Connection header received
0928 02:18:44.632845 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:44.632849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:44.632909 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:44.632989 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:44.633743 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:44.634468 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:44.635667 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:44.636104 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:44.636199 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:44.640894 (+ 4695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.194:47151: BlockingRecv error: recv got EOF from 127.4.140.194:47151 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 02:18:44 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:44 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:44.994681 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:44.985840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.195:52169)
0928 02:18:44.986373 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:44.986388 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:44.986500 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:44.986811 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:44.986814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:44.987185 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:44.987343 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:44.987351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:44.988423 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:44.988428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:44.989633 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:44.989650 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:44.990490 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:44.990494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:44.990646 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:44.991222 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:44.991238 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:44.994422 (+ 3184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":288,"threads_started":1}
W20260928 02:18:44.994870 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:44.985921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:52169 (local address 127.4.140.254:36541)
0928 02:18:44.986074 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:44.986079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:44.986456 (+ 377us) server_negotiation.cc:407] Connection header received
0928 02:18:44.986591 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:44.986595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:44.986669 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:44.986752 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:44.987457 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:44.988282 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:44.989759 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:44.990333 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:44.990428 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:44.994630 (+ 4202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.195:52169: BlockingRecv error: recv got EOF from 127.4.140.195:52169 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 02:18:45 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:45 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:45.284898 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:45.276000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.193:52715)
0928 02:18:45.276447 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:45.276464 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:45.276604 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:45.276905 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:45.276909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:45.277281 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:45.277423 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:45.277432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:45.278377 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:45.278380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:45.279288 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:45.279298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:45.280123 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:45.280130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:45.280314 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:45.280986 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:45.281011 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:45.284522 (+ 3511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":264,"threads_started":1}
W20260928 02:18:45.284945 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:45.276165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:52715 (local address 127.4.140.254:36541)
0928 02:18:45.276382 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:45.276388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:45.276540 (+ 152us) server_negotiation.cc:407] Connection header received
0928 02:18:45.276700 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:45.276703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:45.276776 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:45.276854 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:45.277541 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:45.278265 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:45.279447 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:45.279965 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:45.280090 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:45.284714 (+ 4624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.193:52715: BlockingRecv error: recv got EOF from 127.4.140.193:52715 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 28 02:18:45 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:45 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:45.651218 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:45.641923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.194:59245)
0928 02:18:45.642459 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:45.642481 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:45.642594 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:45.642907 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:45.642910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:45.643288 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:45.643507 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:45.643522 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:45.645010 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:45.645018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:45.646023 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:45.646035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:45.646828 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:45.646833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:45.646953 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:45.647715 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:45.647732 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:45.650976 (+ 3244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":295,"threads_started":1}
W20260928 02:18:45.651379 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:45.642037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:59245 (local address 127.4.140.254:36541)
0928 02:18:45.642229 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:45.642233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:45.642566 (+ 333us) server_negotiation.cc:407] Connection header received
0928 02:18:45.642704 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:45.642707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:45.642770 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:45.642855 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:45.643690 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:45.644854 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:45.646153 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:45.646707 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:45.646799 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:45.651165 (+ 4366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.194:59245: BlockingRecv error: recv got EOF from 127.4.140.194:59245 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 28 02:18:45 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:45 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.195@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:46.004401 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:45.995711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.195:40835)
0928 02:18:45.996189 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:45.996204 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:45.996355 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:45.996689 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:45.996696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:45.997166 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:45.997306 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:45.997314 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:45.998321 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:45.998327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:45.999218 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:45.999228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:45.999949 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:45.999955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:46.000125 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:46.000695 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:46.000712 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:46.004049 (+ 3337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":272,"threads_started":1}
W20260928 02:18:46.004478 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:45.995778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:40835 (local address 127.4.140.254:36541)
0928 02:18:45.996017 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:45.996025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:45.996287 (+ 262us) server_negotiation.cc:407] Connection header received
0928 02:18:45.996475 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:45.996480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:45.996550 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:45.996651 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:45.997448 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:45.998185 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:45.999363 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:45.999772 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:45.999905 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:46.004259 (+ 4354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.195:40835: BlockingRecv error: recv got EOF from 127.4.140.195:40835 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 02:18:46 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:46 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.193@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:46.294821 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:46.286185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.193:50679)
0928 02:18:46.286808 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:46.286824 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:46.286935 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:46.287253 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:46.287257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:46.287610 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:46.287790 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:46.287799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:46.288809 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:46.288812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:46.289718 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:46.289728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:46.290406 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:46.290410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:46.290543 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:46.291145 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:46.291162 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:46.294554 (+ 3392us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":268,"threads_started":1}
W20260928 02:18:46.295018 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:46.286231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:50679 (local address 127.4.140.254:36541)
0928 02:18:46.286481 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:46.286486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:46.286902 (+ 416us) server_negotiation.cc:407] Connection header received
0928 02:18:46.287030 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:46.287033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:46.287094 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:46.287177 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:46.287937 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:46.288680 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:46.289858 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:46.290281 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:46.290373 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:46.294735 (+ 4362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.193:50679: BlockingRecv error: recv got EOF from 127.4.140.193:50679 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Sep 28 02:18:46 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:18:46 dist-test-slave-h37r krb5kdc[12892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.140.194@KRBTEST.COM for oryx/127.4.140.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:18:46.661389 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:46.652411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36541 (local address 127.4.140.194:39513)
0928 02:18:46.652929 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:46.652948 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:46.653111 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:46.653452 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:46.653457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:46.653876 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:46.654053 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:46.654063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:46.655068 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:46.655076 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:46.656038 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:46.656049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:46.656753 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:46.656759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:46.656943 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:46.657521 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:46.657538 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:46.661092 (+ 3554us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.140.254:36541: Server oryx/127.4.140.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":284,"threads_started":1}
W20260928 02:18:46.661628 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:46.652503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:39513 (local address 127.4.140.254:36541)
0928 02:18:46.652735 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:46.652742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:46.653046 (+ 304us) server_negotiation.cc:407] Connection header received
0928 02:18:46.653256 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:46.653260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:46.653320 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:46.653423 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:46.654202 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:46.654912 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:46.656185 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:46.656601 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:46.656732 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:46.661357 (+ 4625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.140.194:39513: BlockingRecv error: recv got EOF from 127.4.140.194:39513 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:18:46.697280 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 13480
I20260928 02:18:46.710143 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:46.698351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.4.140.193:41269)
0928 02:18:46.698959 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:46.698967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:46.698998 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:18:46.699072 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:46.699079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:46.699335 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:46.699508 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:46.701042 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:46.702149 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:46.703474 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:46.704162 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:46.704311 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:46.706878 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:46.706891 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:46.706919 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:46.706972 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:46.709628 (+ 2656us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.140.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 02:18:46.709916 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37748: Request ticket server kudu/127.4.140.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":290,"threads_started":1}
W20260928 02:18:46.710266 13944 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37748: Request ticket server kudu/127.4.140.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 02:18:46.710305 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 13480: Not authorized: Client connection negotiation failed: client connection to 127.4.140.193:41269: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.140.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 02:18:46.710491 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 13480
I20260928 02:18:46.728138 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 13632
I20260928 02:18:46.740845 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:46.729191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38474 (local address 127.4.140.194:36167)
0928 02:18:46.729812 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:46.729819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:46.729848 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:18:46.729918 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:46.729925 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:46.730208 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:46.730430 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:46.731847 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:46.733072 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:46.734274 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:46.734883 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:46.735020 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:46.737616 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:46.737625 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:46.737641 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:46.737681 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:46.740148 (+ 2467us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.140.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 02:18:46.740483 (+ 335us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38474: Request ticket server kudu/127.4.140.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":276,"threads_started":1}
W20260928 02:18:46.740840 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 13632: Not authorized: Client connection negotiation failed: client connection to 127.4.140.194:36167: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.140.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 02:18:46.740949 13945 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38474: Request ticket server kudu/127.4.140.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 02:18:46.741003 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 13632
I20260928 02:18:46.755111 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 13765
I20260928 02:18:46.769783 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:46.756567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35608 (local address 127.4.140.195:36011)
0928 02:18:46.757402 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:46.757412 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:46.757440 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:18:46.757519 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:46.757528 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:46.757907 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:46.758148 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:46.760267 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:46.761878 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:46.763383 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:46.764049 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:46.764174 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:46.766497 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:46.766505 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:46.766519 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:46.766557 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:46.769240 (+ 2683us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.140.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 02:18:46.769553 (+ 313us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35608: Request ticket server kudu/127.4.140.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":510,"threads_started":1}
W20260928 02:18:46.769905 13946 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35608: Request ticket server kudu/127.4.140.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 02:18:46.769984 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 13765: Not authorized: Client connection negotiation failed: client connection to 127.4.140.195:36011: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.140.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 02:18:46.770135 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 13765
I20260928 02:18:46.786651 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 13411
I20260928 02:18:46.787990 13473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:46.903996 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 13411
2026-09-28T02:18:46Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21317 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T02:18:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:18:46Z Disabled control of system clock
I20260928 02:18:46.978994 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:45749
--webserver_interface=127.4.140.254
--webserver_port=0
--builtin_ntp_servers=127.4.140.212:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:45749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:18:47.312896 13951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:47.313207 13951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:47.313285 13951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:47.320544 13951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:18:47.320648 13951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:47.320684 13951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:47.320737 13951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:18:47.320780 13951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:18:47.329939 13951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:45749
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:45749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:47.331215 13951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:47.332940 13951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:47.345141 13957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:47.345122 13959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:47.345093 13956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:47.345614 13951 server_base.cc:1034] running on GCE node
I20260928 02:18:47.346582 13951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:47.348001 13951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:47.349206 13951 hybrid_clock.cc:648] HybridClock initialized: now 1790561927349156 us; error 41 us; skew 500 ppm
I20260928 02:18:47.349561 13951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:18:47.351745 13951 webserver.cc:460] Webserver started at http://127.4.140.254:44541/ using document root <none> and password file <none>
I20260928 02:18:47.352553 13951 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:47.352643 13951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:47.352928 13951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:47.355728 13951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "c7c532973ccd419ea8c1ea093debb330"
format_stamp: "Formatted at 2026-09-28 02:18:47 on dist-test-slave-h37r"
server_key: "477ed3c2dbed590bff632edfe7665410"
server_key_iv: "877b36834c35b6c7dcf41fbfe1dd8799"
server_key_version: "encryptionkey@0"
I20260928 02:18:47.356508 13951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "c7c532973ccd419ea8c1ea093debb330"
format_stamp: "Formatted at 2026-09-28 02:18:47 on dist-test-slave-h37r"
server_key: "477ed3c2dbed590bff632edfe7665410"
server_key_iv: "877b36834c35b6c7dcf41fbfe1dd8799"
server_key_version: "encryptionkey@0"
I20260928 02:18:47.361474 13951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 02:18:47.364843 13965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:47.366300 13951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:18:47.366449 13951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "c7c532973ccd419ea8c1ea093debb330"
format_stamp: "Formatted at 2026-09-28 02:18:47 on dist-test-slave-h37r"
server_key: "477ed3c2dbed590bff632edfe7665410"
server_key_iv: "877b36834c35b6c7dcf41fbfe1dd8799"
server_key_version: "encryptionkey@0"
I20260928 02:18:47.366627 13951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:47.401924 13951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:47.402765 13951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:47.403002 13951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:47.421129 13951 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:45749
I20260928 02:18:47.421165 14016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:45749 every 8 connection(s)
I20260928 02:18:47.422510 13951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:18:47.426267 14017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:47.430495 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 13951
I20260928 02:18:47.430734 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:18:47.431352 4659 external_mini_cluster.cc:1428] Setting key 6d54f9e8f1c77321d54904f5cd4c7e3a
I20260928 02:18:47.437264 14017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330: Bootstrap starting.
I20260928 02:18:47.440856 14017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:47.442081 14017 log.cc:824] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:47.445493 14017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330: No bootstrap required, opened a new log
I20260928 02:18:47.445783 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:47.433534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55502 (local address 127.4.140.254:45749)
0928 02:18:47.434589 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:47.434601 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:47.434653 (+ 52us) server_negotiation.cc:407] Connection header received
0928 02:18:47.436423 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:47.436446 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:47.436924 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:47.437261 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:47.437942 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:47.439224 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:47.441120 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:47.441828 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:47.441974 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:47.443273 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:47.443286 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:47.443313 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:18:47.444000 (+ 687us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:47.444077 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:18:47.444102 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:47.444235 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:47.444508 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 02:18:47.444819 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":386,"threads_started":1}
I20260928 02:18:47.453594 14017 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c532973ccd419ea8c1ea093debb330" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45749 } }
I20260928 02:18:47.454099 14017 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:47.454206 14017 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c532973ccd419ea8c1ea093debb330, State: Initialized, Role: FOLLOWER
I20260928 02:18:47.454850 14017 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c532973ccd419ea8c1ea093debb330" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45749 } }
I20260928 02:18:47.455075 14017 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:18:47.455200 14017 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:18:47.455399 14017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:47.457626 14017 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c532973ccd419ea8c1ea093debb330" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45749 } }
I20260928 02:18:47.458138 14017 leader_election.cc:302] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7c532973ccd419ea8c1ea093debb330; no voters:
I20260928 02:18:47.458752 14017 leader_election.cc:288] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:18:47.458879 14022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:18:47.460000 14022 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330 [term 1 LEADER]: Becoming Leader. State: Replica: c7c532973ccd419ea8c1ea093debb330, State: Running, Role: LEADER
I20260928 02:18:47.460677 14022 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c532973ccd419ea8c1ea093debb330" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45749 } }
I20260928 02:18:47.461068 14017 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:18:47.463222 14024 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7c532973ccd419ea8c1ea093debb330. Latest consensus state: current_term: 1 leader_uuid: "c7c532973ccd419ea8c1ea093debb330" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c532973ccd419ea8c1ea093debb330" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45749 } } }
I20260928 02:18:47.463599 14024 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:47.464169 14023 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7c532973ccd419ea8c1ea093debb330" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c532973ccd419ea8c1ea093debb330" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45749 } } }
I20260928 02:18:47.464490 14023 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:47.464561 14029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:18:47.477443 14029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:18:47.488405 14029 catalog_manager.cc:1347] Generated new cluster ID: 390fc8e2c0694fb489dd06ee684f9130
I20260928 02:18:47.488533 14029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:18:47.508172 14029 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:18:47.509485 14029 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:18:47.518811 14029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7c532973ccd419ea8c1ea093debb330: Generated new TSK 0
I20260928 02:18:47.519757 14029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:18:47.533000 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:45749
--builtin_ntp_servers=127.4.140.212:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 02:18:47.879098 14042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:47.879390 14042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:47.879475 14042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:47.887126 14042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:47.887238 14042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:47.887341 14042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:18:47.896777 14042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:45749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:47.898062 14042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:47.899830 14042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:47.913425 14047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:47.913501 14050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:47.913516 14048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:47.914485 14042 server_base.cc:1034] running on GCE node
I20260928 02:18:47.915093 14042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:47.916121 14042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:47.917330 14042 hybrid_clock.cc:648] HybridClock initialized: now 1790561927917272 us; error 46 us; skew 500 ppm
I20260928 02:18:47.917673 14042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:18:47.927765 14042 webserver.cc:460] Webserver started at http://127.4.140.193:36271/ using document root <none> and password file <none>
I20260928 02:18:47.928596 14042 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:47.928709 14042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:47.929085 14042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:47.932137 14042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "b220b7fb7d504f58b649453380a9bb5c"
format_stamp: "Formatted at 2026-09-28 02:18:47 on dist-test-slave-h37r"
server_key: "81ac24a60c0a6929dfe67cc9dee398f5"
server_key_iv: "3cb889cb9cc2a367e433ca3be862e983"
server_key_version: "encryptionkey@0"
I20260928 02:18:47.932972 14042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "b220b7fb7d504f58b649453380a9bb5c"
format_stamp: "Formatted at 2026-09-28 02:18:47 on dist-test-slave-h37r"
server_key: "81ac24a60c0a6929dfe67cc9dee398f5"
server_key_iv: "3cb889cb9cc2a367e433ca3be862e983"
server_key_version: "encryptionkey@0"
I20260928 02:18:47.938349 14042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:18:47.941948 14056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:47.943462 14042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:18:47.943699 14042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "b220b7fb7d504f58b649453380a9bb5c"
format_stamp: "Formatted at 2026-09-28 02:18:47 on dist-test-slave-h37r"
server_key: "81ac24a60c0a6929dfe67cc9dee398f5"
server_key_iv: "3cb889cb9cc2a367e433ca3be862e983"
server_key_version: "encryptionkey@0"
I20260928 02:18:47.943957 14042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:47.964144 14042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:47.965078 14042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:47.965389 14042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:47.966401 14042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:47.967882 14042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:47.968053 14042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:47.968181 14042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:47.968258 14042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:48.002022 14042 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:34277
I20260928 02:18:48.002086 14168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:34277 every 8 connection(s)
I20260928 02:18:48.003660 14042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:18:48.010762 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 14042
I20260928 02:18:48.011010 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:18:48.011832 4659 external_mini_cluster.cc:1428] Setting key ab860e8c26204303f5cc56e3f4c9b2df
I20260928 02:18:48.014935 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:45749
--builtin_ntp_servers=127.4.140.212:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:18:48.017737 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:48.006539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:52853 (local address 127.4.140.254:45749)
0928 02:18:48.007035 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:48.007042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:48.008441 (+ 1399us) server_negotiation.cc:407] Connection header received
0928 02:18:48.010160 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:48.010165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:48.010268 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:48.010407 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:48.011568 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:48.012669 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:48.014514 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:48.015208 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:48.015364 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:48.016725 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:48.016734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:48.016742 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:18:48.016767 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:48.016822 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:18:48.016844 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:48.016958 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:48.017192 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 02:18:48.017297 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":213,"threads_started":1}
I20260928 02:18:48.018625 14172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:48.007246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:45749 (local address 127.4.140.193:52853)
0928 02:18:48.008274 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:48.008317 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:48.009886 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:48.010608 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:48.010617 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:48.010706 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:48.011384 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:48.011401 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:48.012837 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:48.012841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:48.014371 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:48.014382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:48.015423 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:48.015429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:48.015633 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:48.016402 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:18:48.016404 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:48.016445 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:18:48.016531 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:48.016980 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:48.016989 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:48.016995 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:18:48.017158 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 02:18:48.017468 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":301,"threads_started":1}
I20260928 02:18:48.020193 14169 heartbeater.cc:347] Connected to a master server at 127.4.140.254:45749
I20260928 02:18:48.020524 14169 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:48.021492 14169 heartbeater.cc:511] Master 127.4.140.254:45749 requested a full tablet report, sending...
I20260928 02:18:48.023674 13982 ts_manager.cc:194] Registered new tserver with Master: b220b7fb7d504f58b649453380a9bb5c (127.4.140.193:34277)
I20260928 02:18:48.025133 13982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.140.193:52853
W20260928 02:18:48.341558 14173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:48.341893 14173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:48.342056 14173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:48.350124 14173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:48.350283 14173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:48.350473 14173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:18:48.360883 14173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:45749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:48.362366 14173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:48.364287 14173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:48.378137 14178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:48.378002 14179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:48.378149 14181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:48.379043 14173 server_base.cc:1034] running on GCE node
I20260928 02:18:48.379704 14173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:48.380849 14173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:48.382086 14173 hybrid_clock.cc:648] HybridClock initialized: now 1790561928382036 us; error 43 us; skew 500 ppm
I20260928 02:18:48.382426 14173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:18:48.384763 14173 webserver.cc:460] Webserver started at http://127.4.140.194:42853/ using document root <none> and password file <none>
I20260928 02:18:48.385510 14173 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:48.385654 14173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:48.385984 14173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:48.389024 14173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "f2d2cd8dd1c94bed983bb35a97611d40"
format_stamp: "Formatted at 2026-09-28 02:18:48 on dist-test-slave-h37r"
server_key: "f898960db97d56ca3e85240235aa9941"
server_key_iv: "77eb582ea4f417ae8bc27cd5bca06901"
server_key_version: "encryptionkey@0"
I20260928 02:18:48.389838 14173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "f2d2cd8dd1c94bed983bb35a97611d40"
format_stamp: "Formatted at 2026-09-28 02:18:48 on dist-test-slave-h37r"
server_key: "f898960db97d56ca3e85240235aa9941"
server_key_iv: "77eb582ea4f417ae8bc27cd5bca06901"
server_key_version: "encryptionkey@0"
I20260928 02:18:48.395140 14173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 02:18:48.398818 14187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:48.400290 14173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:18:48.400480 14173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "f2d2cd8dd1c94bed983bb35a97611d40"
format_stamp: "Formatted at 2026-09-28 02:18:48 on dist-test-slave-h37r"
server_key: "f898960db97d56ca3e85240235aa9941"
server_key_iv: "77eb582ea4f417ae8bc27cd5bca06901"
server_key_version: "encryptionkey@0"
I20260928 02:18:48.400738 14173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:48.418601 14173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:48.419471 14173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:48.419786 14173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:48.420864 14173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:48.422353 14173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:48.422477 14173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:48.422614 14173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:48.422704 14173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:48.457309 14173 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:39321
I20260928 02:18:48.457370 14299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:39321 every 8 connection(s)
I20260928 02:18:48.458663 14173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:18:48.459170 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 14173
I20260928 02:18:48.459388 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:18:48.460076 4659 external_mini_cluster.cc:1428] Setting key d2b2bc2793577ce014af0e281f80b36b
I20260928 02:18:48.463522 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:45749
--builtin_ntp_servers=127.4.140.212:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:18:48.474330 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:48.461497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:52883 (local address 127.4.140.254:45749)
0928 02:18:48.461663 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:48.461668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:48.463568 (+ 1900us) server_negotiation.cc:407] Connection header received
0928 02:18:48.466483 (+ 2915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:48.466487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:48.466558 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:48.466663 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:48.468134 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:48.468822 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:48.471384 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:48.471782 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:48.471884 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:48.473624 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:48.473629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:48.473632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:18:48.473656 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:48.473687 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:18:48.473698 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:48.473770 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:48.474029 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 02:18:48.474106 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:18:48.475924 14302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:48.462217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:45749 (local address 127.4.140.194:52883)
0928 02:18:48.463386 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:48.463433 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:48.466168 (+ 2735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:48.466937 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:48.466952 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:48.467087 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:48.467962 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:48.467991 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:48.468978 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:48.468986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:48.471204 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:48.471228 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:48.472068 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:48.472076 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:48.472427 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:48.473243 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:18:48.473249 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:48.473312 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:18:48.473422 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:48.473797 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:48.473809 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:48.473821 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:18:48.474002 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 02:18:48.474474 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":360,"threads_started":1}
I20260928 02:18:48.477347 14300 heartbeater.cc:347] Connected to a master server at 127.4.140.254:45749
I20260928 02:18:48.477674 14300 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:48.478451 14300 heartbeater.cc:511] Master 127.4.140.254:45749 requested a full tablet report, sending...
I20260928 02:18:48.479836 13982 ts_manager.cc:194] Registered new tserver with Master: f2d2cd8dd1c94bed983bb35a97611d40 (127.4.140.194:39321)
I20260928 02:18:48.480751 13982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.140.194:52883
W20260928 02:18:48.797766 14303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:48.798035 14303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:48.798206 14303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:48.805480 14303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:48.805598 14303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:48.805733 14303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:18:48.814546 14303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:45749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:48.815814 14303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:48.817605 14303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:48.830410 14311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:48.830492 14309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:48.830482 14308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:48.833415 14303 server_base.cc:1034] running on GCE node
I20260928 02:18:48.834015 14303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:48.834921 14303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:48.836139 14303 hybrid_clock.cc:648] HybridClock initialized: now 1790561928836091 us; error 60 us; skew 500 ppm
I20260928 02:18:48.836459 14303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:18:48.838622 14303 webserver.cc:460] Webserver started at http://127.4.140.195:46229/ using document root <none> and password file <none>
I20260928 02:18:48.839298 14303 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:48.839422 14303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:48.839737 14303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:48.842564 14303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "08d0f40253e54ef487d00ad472270dc6"
format_stamp: "Formatted at 2026-09-28 02:18:48 on dist-test-slave-h37r"
server_key: "50596ceda57185866cb3aedee9d91be8"
server_key_iv: "e31f1b41b2b837f0699b030bb20e6358"
server_key_version: "encryptionkey@0"
I20260928 02:18:48.843355 14303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "08d0f40253e54ef487d00ad472270dc6"
format_stamp: "Formatted at 2026-09-28 02:18:48 on dist-test-slave-h37r"
server_key: "50596ceda57185866cb3aedee9d91be8"
server_key_iv: "e31f1b41b2b837f0699b030bb20e6358"
server_key_version: "encryptionkey@0"
I20260928 02:18:48.848505 14303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260928 02:18:48.852140 14317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:48.853502 14303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:18:48.853672 14303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "08d0f40253e54ef487d00ad472270dc6"
format_stamp: "Formatted at 2026-09-28 02:18:48 on dist-test-slave-h37r"
server_key: "50596ceda57185866cb3aedee9d91be8"
server_key_iv: "e31f1b41b2b837f0699b030bb20e6358"
server_key_version: "encryptionkey@0"
I20260928 02:18:48.853915 14303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:48.878410 14303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:48.879324 14303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:48.879631 14303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:48.880713 14303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:48.882210 14303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:48.882318 14303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:48.882445 14303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:48.882537 14303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:48.914429 14303 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:35281
I20260928 02:18:48.914518 14429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:35281 every 8 connection(s)
I20260928 02:18:48.915719 14303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:18:48.920006 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 14303
I20260928 02:18:48.920187 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:18:48.920601 4659 external_mini_cluster.cc:1428] Setting key 7a7346c78f5bafac469984f4c3f331c2
I20260928 02:18:48.928516 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:48.918304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:57079 (local address 127.4.140.254:45749)
0928 02:18:48.918483 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:48.918487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:48.919848 (+ 1361us) server_negotiation.cc:407] Connection header received
0928 02:18:48.922230 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:48.922235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:48.922348 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:48.922462 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:48.923857 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:48.924649 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:48.926129 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:48.926532 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:48.926631 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:48.927827 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:48.927832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:48.927835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:18:48.927852 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:48.927881 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:18:48.927939 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:48.928017 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:48.928222 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 02:18:48.928292 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:18:48.929486 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:48.918775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:45749 (local address 127.4.140.195:57079)
0928 02:18:48.919678 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:48.919717 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:48.921940 (+ 2223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:48.922729 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:48.922740 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:48.922872 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:48.923706 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:48.923726 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:48.924782 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:48.924786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:48.926004 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:48.926015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:48.926717 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:48.926721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:48.926908 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:48.927543 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:18:48.927545 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:48.927583 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:18:48.927658 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:48.928029 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:48.928036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:48.928042 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:18:48.928191 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 02:18:48.928473 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":253,"threads_started":1}
I20260928 02:18:48.930800 14430 heartbeater.cc:347] Connected to a master server at 127.4.140.254:45749
I20260928 02:18:48.931047 14430 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:48.931715 14430 heartbeater.cc:511] Master 127.4.140.254:45749 requested a full tablet report, sending...
I20260928 02:18:48.933032 13982 ts_manager.cc:194] Registered new tserver with Master: 08d0f40253e54ef487d00ad472270dc6 (127.4.140.195:35281)
I20260928 02:18:48.933765 13982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.140.195:57079
I20260928 02:18:48.935824 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:18:48.941534 14171 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:48.940116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55518 (local address 127.4.140.254:45749)
0928 02:18:48.940367 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:48.940375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:48.940678 (+ 303us) server_negotiation.cc:407] Connection header received
0928 02:18:48.940801 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:48.940804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:48.940871 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:48.940948 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:48.941320 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55518: BlockingRecv error: recv got EOF from 127.0.0.1:55518 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:18:48.941939 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 14042
I20260928 02:18:48.949501 14439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:48.942927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.4.140.193:34277)
0928 02:18:48.943477 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:48.943484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:48.943508 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:18:48.943559 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:48.943564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:48.943759 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:48.943876 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:48.944271 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:48.945239 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:48.946425 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:48.946947 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:48.947056 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:48.948139 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:48.948146 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:48.948157 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:18:48.948779 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:48.948824 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:18:48.948842 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:48.948929 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:48.949094 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 02:18:48.949226 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":301,"threads_started":1}
I20260928 02:18:48.950659 14164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:49.029642 14169 heartbeater.cc:503] Master 127.4.140.254:45749 was elected leader, sending a full tablet report...
I20260928 02:18:49.031124 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 14042
W20260928 02:18:49.046867 13968 connection.cc:535] server connection from 127.4.140.193:52853 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:18:49.047017 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 14173
I20260928 02:18:49.055109 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:49.047989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49588 (local address 127.4.140.194:39321)
0928 02:18:49.048573 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:49.048580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:49.048617 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:18:49.048676 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:49.048683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:49.048929 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:49.049129 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:49.049555 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:49.050544 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:49.051788 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:49.052529 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:49.052672 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:49.053686 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:49.053696 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:49.053709 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:18:49.054319 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:49.054365 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:18:49.054384 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:49.054521 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:49.054629 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 02:18:49.054772 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":280,"threads_started":1}
I20260928 02:18:49.056473 14295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:49.138006 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 14173
I20260928 02:18:49.154686 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 14303
I20260928 02:18:49.162712 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:49.155596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.4.140.195:35281)
0928 02:18:49.156198 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:49.156205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:49.156235 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:18:49.156291 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:49.156297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:49.156586 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:49.156735 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:49.157115 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:49.158179 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:49.159410 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:49.160047 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:49.160172 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:49.161191 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:49.161199 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:49.161213 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:18:49.161872 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:49.161934 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:18:49.161954 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:49.162070 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:49.162214 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 02:18:49.162347 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":321,"threads_started":1}
I20260928 02:18:49.164005 14425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:49.245306 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 14303
I20260928 02:18:49.261205 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 13951
I20260928 02:18:49.262491 14012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:49.364246 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 13951
2026-09-28T02:18:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2456 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T02:18:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:18:49Z Disabled control of system clock
I20260928 02:18:49.436180 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:34491
--webserver_interface=127.4.140.254
--webserver_port=0
--builtin_ntp_servers=127.4.140.212:43227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:34491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:18:49.766053 14449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:49.766343 14449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:49.766426 14449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:49.773571 14449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:18:49.773686 14449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:49.773725 14449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:49.773782 14449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:18:49.773828 14449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:18:49.783035 14449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:43227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:34491
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:34491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:49.784372 14449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:49.786101 14449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:49.797631 14454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:49.797770 14457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:49.797726 14455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:49.798620 14449 server_base.cc:1034] running on GCE node
I20260928 02:18:49.799450 14449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:49.800953 14449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:49.802183 14449 hybrid_clock.cc:648] HybridClock initialized: now 1790561929802143 us; error 59 us; skew 500 ppm
I20260928 02:18:49.802542 14449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:18:49.804857 14449 webserver.cc:460] Webserver started at http://127.4.140.254:37781/ using document root <none> and password file <none>
I20260928 02:18:49.805608 14449 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:49.805742 14449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:49.806118 14449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:49.809144 14449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "0e81bbbb99dc4363aaa1b3a12810a75a"
format_stamp: "Formatted at 2026-09-28 02:18:49 on dist-test-slave-h37r"
server_key: "b7501b8bd61b3a4fc6564d7cf848376f"
server_key_iv: "0699056a059e5ac2ca6b2123f1d9a278"
server_key_version: "encryptionkey@0"
I20260928 02:18:49.809943 14449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "0e81bbbb99dc4363aaa1b3a12810a75a"
format_stamp: "Formatted at 2026-09-28 02:18:49 on dist-test-slave-h37r"
server_key: "b7501b8bd61b3a4fc6564d7cf848376f"
server_key_iv: "0699056a059e5ac2ca6b2123f1d9a278"
server_key_version: "encryptionkey@0"
I20260928 02:18:49.815186 14449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:18:49.818827 14463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:49.820277 14449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:18:49.820466 14449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "0e81bbbb99dc4363aaa1b3a12810a75a"
format_stamp: "Formatted at 2026-09-28 02:18:49 on dist-test-slave-h37r"
server_key: "b7501b8bd61b3a4fc6564d7cf848376f"
server_key_iv: "0699056a059e5ac2ca6b2123f1d9a278"
server_key_version: "encryptionkey@0"
I20260928 02:18:49.820701 14449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:49.826512 14449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:49.827322 14449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:49.827626 14449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:49.845865 14449 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:34491
I20260928 02:18:49.845888 14514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:34491 every 8 connection(s)
I20260928 02:18:49.847266 14449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:18:49.851101 14515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:49.857435 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 14449
I20260928 02:18:49.857631 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:18:49.858502 4659 external_mini_cluster.cc:1428] Setting key 9d7a31a1fc311065ec7c6756d2621d45
I20260928 02:18:49.862846 14515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a: Bootstrap starting.
I20260928 02:18:49.866068 14515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:49.867111 14515 log.cc:824] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:49.867359 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:49.860659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44262 (local address 127.4.140.254:34491)
0928 02:18:49.861728 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:49.861743 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:49.861796 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:18:49.863741 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:49.863763 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:49.864252 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:49.864655 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:49.865712 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:49.865718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:49.865731 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:18:49.865744 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:49.865797 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:18:49.865815 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:49.866011 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:49.866151 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 02:18:49.866436 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":298,"threads_started":1}
I20260928 02:18:49.870189 14515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a: No bootstrap required, opened a new log
I20260928 02:18:49.877562 14515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e81bbbb99dc4363aaa1b3a12810a75a" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 34491 } }
I20260928 02:18:49.877945 14515 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:49.878042 14515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e81bbbb99dc4363aaa1b3a12810a75a, State: Initialized, Role: FOLLOWER
I20260928 02:18:49.878640 14515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e81bbbb99dc4363aaa1b3a12810a75a" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 34491 } }
I20260928 02:18:49.878870 14515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:18:49.878999 14515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:18:49.879186 14515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:49.881417 14515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e81bbbb99dc4363aaa1b3a12810a75a" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 34491 } }
I20260928 02:18:49.881946 14515 leader_election.cc:302] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e81bbbb99dc4363aaa1b3a12810a75a; no voters:
I20260928 02:18:49.882550 14515 leader_election.cc:288] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:18:49.882720 14520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:18:49.883435 14520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a [term 1 LEADER]: Becoming Leader. State: Replica: 0e81bbbb99dc4363aaa1b3a12810a75a, State: Running, Role: LEADER
I20260928 02:18:49.884112 14520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e81bbbb99dc4363aaa1b3a12810a75a" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 34491 } }
I20260928 02:18:49.884822 14515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:18:49.887387 14521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e81bbbb99dc4363aaa1b3a12810a75a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e81bbbb99dc4363aaa1b3a12810a75a" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 34491 } } }
I20260928 02:18:49.887812 14521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:49.887632 14522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e81bbbb99dc4363aaa1b3a12810a75a. Latest consensus state: current_term: 1 leader_uuid: "0e81bbbb99dc4363aaa1b3a12810a75a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e81bbbb99dc4363aaa1b3a12810a75a" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 34491 } } }
I20260928 02:18:49.888054 14522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:49.888861 14527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:18:49.894819 14527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:18:49.904932 14527 catalog_manager.cc:1347] Generated new cluster ID: 9087378155fe4aec97796d940b59ce29
I20260928 02:18:49.905063 14527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:18:49.939827 14527 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:18:49.940423 14527 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:18:49.946970 14527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e81bbbb99dc4363aaa1b3a12810a75a: Generated new TSK 0
I20260928 02:18:49.948028 14527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:18:49.964078 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:34491
--builtin_ntp_servers=127.4.140.212:43227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:18:50.298533 14539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:50.298885 14539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:50.299072 14539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:50.307240 14539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:50.307389 14539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:50.307607 14539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:18:50.317726 14539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:43227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:34491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:50.319119 14539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:50.320981 14539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:50.334636 14545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:50.334699 14547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:50.335379 14544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:50.336190 14539 server_base.cc:1034] running on GCE node
I20260928 02:18:50.336839 14539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:50.337823 14539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:50.339025 14539 hybrid_clock.cc:648] HybridClock initialized: now 1790561930338957 us; error 57 us; skew 500 ppm
I20260928 02:18:50.339350 14539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:18:50.341785 14539 webserver.cc:460] Webserver started at http://127.4.140.193:38817/ using document root <none> and password file <none>
I20260928 02:18:50.342504 14539 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:50.342602 14539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:50.342967 14539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:50.345858 14539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "8d9c3a3175fc46cd84656837036da199"
format_stamp: "Formatted at 2026-09-28 02:18:50 on dist-test-slave-h37r"
server_key: "484dfae548523313ef0d8742789f4ab6"
server_key_iv: "edcf4d8787ee32ab5486820b29b36a6f"
server_key_version: "encryptionkey@0"
I20260928 02:18:50.346648 14539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "8d9c3a3175fc46cd84656837036da199"
format_stamp: "Formatted at 2026-09-28 02:18:50 on dist-test-slave-h37r"
server_key: "484dfae548523313ef0d8742789f4ab6"
server_key_iv: "edcf4d8787ee32ab5486820b29b36a6f"
server_key_version: "encryptionkey@0"
I20260928 02:18:50.351836 14539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 02:18:50.355253 14553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:50.356654 14539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:18:50.356824 14539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "8d9c3a3175fc46cd84656837036da199"
format_stamp: "Formatted at 2026-09-28 02:18:50 on dist-test-slave-h37r"
server_key: "484dfae548523313ef0d8742789f4ab6"
server_key_iv: "edcf4d8787ee32ab5486820b29b36a6f"
server_key_version: "encryptionkey@0"
I20260928 02:18:50.357040 14539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:50.362788 14539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:50.363577 14539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:50.363842 14539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:50.364758 14539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:50.366562 14539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:50.366691 14539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:50.366817 14539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:50.366901 14539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:50.399864 14539 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:36365
I20260928 02:18:50.399978 14665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:36365 every 8 connection(s)
I20260928 02:18:50.401242 14539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:18:50.406486 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 14539
I20260928 02:18:50.406682 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:18:50.407289 4659 external_mini_cluster.cc:1428] Setting key 6267d0cf62781939c527ad6852b5609c
I20260928 02:18:50.409643 14668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:50.403779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:60913 (local address 127.4.140.254:34491)
0928 02:18:50.404222 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:50.404227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:50.405403 (+ 1176us) server_negotiation.cc:407] Connection header received
0928 02:18:50.407140 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:50.407146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:50.407211 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:50.407335 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:50.408800 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:50.408804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:50.408807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:18:50.408843 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:50.408897 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:18:50.408909 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:50.409038 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:50.409304 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 02:18:50.409368 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":253,"threads_started":1}
I20260928 02:18:50.410156 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:34491
--builtin_ntp_servers=127.4.140.212:43227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 02:18:50.410622 14669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:50.404358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:34491 (local address 127.4.140.193:60913)
0928 02:18:50.405233 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:50.405271 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:50.406892 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:50.407565 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:50.407573 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:50.407660 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:50.408392 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:18:50.408396 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:50.408448 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:18:50.408553 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:50.409091 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:50.409101 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:50.409109 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:18:50.409382 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 02:18:50.409626 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":262,"threads_started":1}
I20260928 02:18:50.412187 14666 heartbeater.cc:347] Connected to a master server at 127.4.140.254:34491
I20260928 02:18:50.412493 14666 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:50.413041 14666 heartbeater.cc:511] Master 127.4.140.254:34491 requested a full tablet report, sending...
I20260928 02:18:50.415197 14480 ts_manager.cc:194] Registered new tserver with Master: 8d9c3a3175fc46cd84656837036da199 (127.4.140.193:36365)
W20260928 02:18:50.738154 14670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:50.738426 14670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:50.738582 14670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:50.745771 14670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:50.745869 14670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:50.746101 14670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:18:50.755605 14670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:43227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:34491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:50.757570 14670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:50.759742 14670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:50.773231 14676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:50.773314 14675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:50.773723 14678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:50.774070 14670 server_base.cc:1034] running on GCE node
I20260928 02:18:50.774682 14670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:50.775563 14670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:50.776786 14670 hybrid_clock.cc:648] HybridClock initialized: now 1790561930776739 us; error 59 us; skew 500 ppm
I20260928 02:18:50.777101 14670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:18:50.779193 14670 webserver.cc:460] Webserver started at http://127.4.140.194:33689/ using document root <none> and password file <none>
I20260928 02:18:50.779874 14670 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:50.780054 14670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:50.780386 14670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:50.783120 14670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "b7172ba2c12e43fd8863a186e692bdde"
format_stamp: "Formatted at 2026-09-28 02:18:50 on dist-test-slave-h37r"
server_key: "1537482658b089e77ebba3cc8aa2297c"
server_key_iv: "f65e7b1c2c415d762854753b668b655a"
server_key_version: "encryptionkey@0"
I20260928 02:18:50.783962 14670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "b7172ba2c12e43fd8863a186e692bdde"
format_stamp: "Formatted at 2026-09-28 02:18:50 on dist-test-slave-h37r"
server_key: "1537482658b089e77ebba3cc8aa2297c"
server_key_iv: "f65e7b1c2c415d762854753b668b655a"
server_key_version: "encryptionkey@0"
I20260928 02:18:50.789105 14670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:18:50.792572 14684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:50.793885 14670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:18:50.794057 14670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "b7172ba2c12e43fd8863a186e692bdde"
format_stamp: "Formatted at 2026-09-28 02:18:50 on dist-test-slave-h37r"
server_key: "1537482658b089e77ebba3cc8aa2297c"
server_key_iv: "f65e7b1c2c415d762854753b668b655a"
server_key_version: "encryptionkey@0"
I20260928 02:18:50.794283 14670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:50.799652 14670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:50.800463 14670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:50.800729 14670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:50.801607 14670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:50.802949 14670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:50.803073 14670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:50.803220 14670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:50.803308 14670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:50.836369 14670 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:46179
I20260928 02:18:50.836416 14796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:46179 every 8 connection(s)
I20260928 02:18:50.837705 14670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:18:50.844108 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 14670
I20260928 02:18:50.844275 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:18:50.844781 4659 external_mini_cluster.cc:1428] Setting key 3f1d620c729aa3cd549189e6a0880356
I20260928 02:18:50.845876 14668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:50.840252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:35297 (local address 127.4.140.254:34491)
0928 02:18:50.840406 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:50.840410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:50.841818 (+ 1408us) server_negotiation.cc:407] Connection header received
0928 02:18:50.843381 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:50.843385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:50.843436 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:50.843521 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:50.845043 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:50.845047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:50.845051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:18:50.845068 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:50.845098 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:18:50.845110 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:50.845245 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:50.845496 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 02:18:50.845564 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:18:50.846956 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:50.840773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:34491 (local address 127.4.140.194:35297)
0928 02:18:50.841671 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:50.841707 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:50.843153 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:50.843740 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:50.843748 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:50.843837 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:50.844672 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:18:50.844674 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:50.844713 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:18:50.844797 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:50.845219 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:50.845231 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:50.845239 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:18:50.845547 (+ 308us) client_negotiation.cc:240] Negotiation successful
0928 02:18:50.845808 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":259,"threads_started":1}
I20260928 02:18:50.847622 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:34491
--builtin_ntp_servers=127.4.140.212:43227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 02:18:50.848626 14797 heartbeater.cc:347] Connected to a master server at 127.4.140.254:34491
I20260928 02:18:50.848954 14797 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:50.849625 14797 heartbeater.cc:511] Master 127.4.140.254:34491 requested a full tablet report, sending...
I20260928 02:18:50.851158 14480 ts_manager.cc:194] Registered new tserver with Master: b7172ba2c12e43fd8863a186e692bdde (127.4.140.194:46179)
W20260928 02:18:51.182435 14800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:51.182714 14800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:51.182914 14800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:51.190600 14800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:51.190732 14800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:51.190940 14800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:18:51.200294 14800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:43227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:34491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:51.201570 14800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:51.203364 14800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:51.217783 14805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:51.217751 14806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:51.217998 14808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:51.219111 14800 server_base.cc:1034] running on GCE node
I20260928 02:18:51.219884 14800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:51.221383 14800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:51.222668 14800 hybrid_clock.cc:648] HybridClock initialized: now 1790561931222541 us; error 106 us; skew 500 ppm
I20260928 02:18:51.223153 14800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:18:51.226325 14800 webserver.cc:460] Webserver started at http://127.4.140.195:37711/ using document root <none> and password file <none>
I20260928 02:18:51.227216 14800 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:51.227346 14800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:51.227798 14800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:51.232383 14800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "c391697bed14449dae9b6f3b3b5f0f06"
format_stamp: "Formatted at 2026-09-28 02:18:51 on dist-test-slave-h37r"
server_key: "ce726c5af33affc182475a8dcc020268"
server_key_iv: "41c2eb2f3230dcfd8ec328ac13910aff"
server_key_version: "encryptionkey@0"
I20260928 02:18:51.233726 14800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "c391697bed14449dae9b6f3b3b5f0f06"
format_stamp: "Formatted at 2026-09-28 02:18:51 on dist-test-slave-h37r"
server_key: "ce726c5af33affc182475a8dcc020268"
server_key_iv: "41c2eb2f3230dcfd8ec328ac13910aff"
server_key_version: "encryptionkey@0"
I20260928 02:18:51.241264 14800 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260928 02:18:51.245322 14814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:51.247165 14800 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:18:51.247382 14800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "c391697bed14449dae9b6f3b3b5f0f06"
format_stamp: "Formatted at 2026-09-28 02:18:51 on dist-test-slave-h37r"
server_key: "ce726c5af33affc182475a8dcc020268"
server_key_iv: "41c2eb2f3230dcfd8ec328ac13910aff"
server_key_version: "encryptionkey@0"
I20260928 02:18:51.247650 14800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:51.255963 14800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:51.256767 14800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:51.257054 14800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:51.258042 14800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:51.259557 14800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:51.259640 14800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:51.259785 14800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:51.259855 14800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:51.294739 14800 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:39875
I20260928 02:18:51.294790 14926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:39875 every 8 connection(s)
I20260928 02:18:51.296139 14800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:18:51.303071 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 14800
I20260928 02:18:51.303251 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:18:51.303683 4659 external_mini_cluster.cc:1428] Setting key e4584670d910d5eba86d70a7e6282842
I20260928 02:18:51.304809 14668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:51.298852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:40073 (local address 127.4.140.254:34491)
0928 02:18:51.299057 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:51.299063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:51.300576 (+ 1513us) server_negotiation.cc:407] Connection header received
0928 02:18:51.302326 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:51.302330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:51.302379 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:51.302464 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:51.303940 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:51.303947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:51.303953 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:18:51.303978 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:51.304021 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:18:51.304035 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:51.304185 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:51.304416 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 02:18:51.304501 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:18:51.305755 14929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:51.299437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:34491 (local address 127.4.140.195:40073)
0928 02:18:51.300410 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:51.300447 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:51.302059 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:51.302673 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:51.302681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:51.302793 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:51.303570 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:18:51.303573 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:51.303612 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:18:51.303719 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:51.304158 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:51.304171 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:51.304179 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:18:51.304485 (+ 306us) client_negotiation.cc:240] Negotiation successful
0928 02:18:51.304744 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":261,"threads_started":1}
I20260928 02:18:51.307350 14927 heartbeater.cc:347] Connected to a master server at 127.4.140.254:34491
I20260928 02:18:51.307674 14927 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:51.308336 14927 heartbeater.cc:511] Master 127.4.140.254:34491 requested a full tablet report, sending...
I20260928 02:18:51.309727 14480 ts_manager.cc:194] Registered new tserver with Master: c391697bed14449dae9b6f3b3b5f0f06 (127.4.140.195:39875)
I20260928 02:18:51.319288 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:18:51.325482 14668 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:18:51.323652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44278 (local address 127.4.140.254:34491)
0928 02:18:51.323823 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:51.323828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:51.324446 (+ 618us) server_negotiation.cc:407] Connection header received
0928 02:18:51.324652 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:51.324657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:51.324717 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:51.324842 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:51.325197 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44278: BlockingRecv error: recv got EOF from 127.0.0.1:44278 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:18:51.325909 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 14539
I20260928 02:18:51.329728 14936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:51.327165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.4.140.193:36365)
0928 02:18:51.327734 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:51.327741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:51.327768 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:18:51.327822 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:51.327828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:51.328130 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:51.328258 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:51.329124 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:51.329130 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:51.329142 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:18:51.329158 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:51.329197 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:18:51.329210 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:51.329311 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:51.329419 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 02:18:51.329500 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":260,"threads_started":1}
I20260928 02:18:51.330909 14661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:51.414057 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 14539
I20260928 02:18:51.430078 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 14670
I20260928 02:18:51.433789 14938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:51.431072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49172 (local address 127.4.140.194:46179)
0928 02:18:51.431592 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:51.431599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:51.431628 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:18:51.431695 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:51.431701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:51.431984 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:51.432123 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:51.433009 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:51.433016 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:51.433027 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:18:51.433055 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:51.433099 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:18:51.433114 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:51.433219 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:51.433384 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 02:18:51.433501 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":243,"threads_started":1}
I20260928 02:18:51.435105 14792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:51.516060 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 14670
I20260928 02:18:51.532653 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 14800
I20260928 02:18:51.536859 14940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:51.533789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48056 (local address 127.4.140.195:39875)
0928 02:18:51.534590 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:51.534597 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:51.534629 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:18:51.534693 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:51.534701 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:51.534924 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:51.535069 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:51.536141 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:51.536148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:51.536161 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:18:51.536173 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:51.536224 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:18:51.536241 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:51.536370 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:51.536417 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 02:18:51.536534 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":456,"threads_started":1}
I20260928 02:18:51.538172 14922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:51.618426 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 14800
I20260928 02:18:51.633988 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 14449
I20260928 02:18:51.635090 14510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:51.734719 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 14449
2026-09-28T02:18:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2369 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:18:51 dist-test-slave-h37r krb5kdc[14945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:18:51 dist-test-slave-h37r krb5kdc[14945](info): set up 2 sockets
Sep 28 02:18:51 dist-test-slave-h37r krb5kdc[14945](info): commencing operation
krb5kdc: starting...
W20260928 02:18:52.836918 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:18:52 dist-test-slave-h37r krb5kdc[14945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:18:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:18:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:18:53.071774 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:39447
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:39447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:53.396168 14958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:53.396474 14958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:53.396634 14958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:53.403947 14958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:18:53.404091 14958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:53.404215 14958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:53.404313 14958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:18:53.404379 14958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:18:53.414220 14958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:39447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:39447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:53.415603 14958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:53.416968 14958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:53.428745 14963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:53.428745 14966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:53.428776 14964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:53.430101 14958 server_base.cc:1034] running on GCE node
I20260928 02:18:53.430922 14958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:53.432540 14958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:53.433779 14958 hybrid_clock.cc:648] HybridClock initialized: now 1790561933433728 us; error 59 us; skew 500 ppm
I20260928 02:18:53.434142 14958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:53 dist-test-slave-h37r krb5kdc[14945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561933, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:53.443593 14958 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:18:53.445246 14958 webserver.cc:460] Webserver started at http://127.4.140.254:39315/ using document root <none> and password file <none>
I20260928 02:18:53.445946 14958 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:53.446048 14958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:53.446310 14958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:53.449352 14958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "1dbe2c94f09b4cd1bb396363c9ed0bd3"
format_stamp: "Formatted at 2026-09-28 02:18:53 on dist-test-slave-h37r"
server_key: "8c7911437d6ae686dd6183f0604ed11c"
server_key_iv: "187691a2f735f3cd145519dca583b7a7"
server_key_version: "encryptionkey@0"
I20260928 02:18:53.450067 14958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "1dbe2c94f09b4cd1bb396363c9ed0bd3"
format_stamp: "Formatted at 2026-09-28 02:18:53 on dist-test-slave-h37r"
server_key: "8c7911437d6ae686dd6183f0604ed11c"
server_key_iv: "187691a2f735f3cd145519dca583b7a7"
server_key_version: "encryptionkey@0"
I20260928 02:18:53.455474 14958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:18:53.459028 14973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:53.460501 14958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:18:53.460691 14958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "1dbe2c94f09b4cd1bb396363c9ed0bd3"
format_stamp: "Formatted at 2026-09-28 02:18:53 on dist-test-slave-h37r"
server_key: "8c7911437d6ae686dd6183f0604ed11c"
server_key_iv: "187691a2f735f3cd145519dca583b7a7"
server_key_version: "encryptionkey@0"
I20260928 02:18:53.460947 14958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:53.486361 14958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:53.490846 14958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:53.491127 14958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:53.509253 14958 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:39447
I20260928 02:18:53.509276 15024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:39447 every 8 connection(s)
I20260928 02:18:53.510695 14958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:18:53.514230 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 14958
I20260928 02:18:53.514458 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:18:53.514809 15025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:53.515313 4659 external_mini_cluster.cc:1428] Setting key a6533b695740ccacf74ba9da4a64fb36
I20260928 02:18:53.527817 15025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3: Bootstrap starting.
Sep 28 02:18:53 dist-test-slave-h37r krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:18:53.531507 15025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:53.532642 15025 log.cc:824] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:53.536031 15025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3: No bootstrap required, opened a new log
I20260928 02:18:53.539168 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:53.517639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51966 (local address 127.4.140.254:39447)
0928 02:18:53.518479 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:53.518496 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:53.518541 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:18:53.520224 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:53.520248 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:53.520681 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:53.521125 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:53.522619 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:53.523945 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:53.525218 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:53.525945 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:53.526164 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:53.530526 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:53.530534 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:53.530548 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:53.530594 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:53.534169 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:53.535068 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:53.535075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:53.535082 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:53.535269 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:53.535708 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:53.535712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:53.535714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:53.536387 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:53.536654 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:53.537201 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 02:18:53.537702 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":382,"threads_started":1}
I20260928 02:18:53.545647 15025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe2c94f09b4cd1bb396363c9ed0bd3" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 39447 } }
I20260928 02:18:53.546134 15025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:53.546253 15025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbe2c94f09b4cd1bb396363c9ed0bd3, State: Initialized, Role: FOLLOWER
I20260928 02:18:53.546890 15025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe2c94f09b4cd1bb396363c9ed0bd3" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 39447 } }
I20260928 02:18:53.547118 15025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:18:53.547269 15025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:18:53.547444 15025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:53.549623 15025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe2c94f09b4cd1bb396363c9ed0bd3" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 39447 } }
I20260928 02:18:53.550117 15025 leader_election.cc:302] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1dbe2c94f09b4cd1bb396363c9ed0bd3; no voters:
I20260928 02:18:53.550784 15025 leader_election.cc:288] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:18:53.551023 15030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:18:53.551870 15030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3 [term 1 LEADER]: Becoming Leader. State: Replica: 1dbe2c94f09b4cd1bb396363c9ed0bd3, State: Running, Role: LEADER
I20260928 02:18:53.552460 15030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe2c94f09b4cd1bb396363c9ed0bd3" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 39447 } }
I20260928 02:18:53.553138 15025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:18:53.555703 15031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1dbe2c94f09b4cd1bb396363c9ed0bd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe2c94f09b4cd1bb396363c9ed0bd3" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 39447 } } }
I20260928 02:18:53.555676 15032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dbe2c94f09b4cd1bb396363c9ed0bd3. Latest consensus state: current_term: 1 leader_uuid: "1dbe2c94f09b4cd1bb396363c9ed0bd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbe2c94f09b4cd1bb396363c9ed0bd3" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 39447 } } }
I20260928 02:18:53.556133 15031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:53.556180 15032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:18:53.558714 15036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:18:53.563273 15036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:18:53.571762 15036 catalog_manager.cc:1347] Generated new cluster ID: 15fa6d037b094f16929db5a26513825f
I20260928 02:18:53.571913 15036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:18:53.580615 15036 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:18:53.581851 15036 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:18:53.591547 15036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3: Generated new TSK 0
I20260928 02:18:53.592553 15036 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
I20260928 02:18:53.649451 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:39447
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:53.985026 15051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:53.985320 15051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:53.985487 15051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:53.993088 15051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:53.993202 15051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:53.993360 15051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:18:54.003270 15051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:39447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:54.004696 15051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:54.006057 15051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:54.019356 15057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:54.019421 15059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:54.019445 15056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:54.020318 15051 server_base.cc:1034] running on GCE node
I20260928 02:18:54.021044 15051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:54.022110 15051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:54.023386 15051 hybrid_clock.cc:648] HybridClock initialized: now 1790561934023325 us; error 47 us; skew 500 ppm
I20260928 02:18:54.023782 15051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:54 dist-test-slave-h37r krb5kdc[14945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561934, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:54.027271 15051 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:18:54.028966 15051 webserver.cc:460] Webserver started at http://127.4.140.193:43605/ using document root <none> and password file <none>
I20260928 02:18:54.029712 15051 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:54.029875 15051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:54.030218 15051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:54.033469 15051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "26837d8717cf4d27a32dcb57fb28a9d9"
format_stamp: "Formatted at 2026-09-28 02:18:54 on dist-test-slave-h37r"
server_key: "b400f2fe2cef26b1830ce03150245a74"
server_key_iv: "b3c717e02b35f8df0d47361f0633f26a"
server_key_version: "encryptionkey@0"
I20260928 02:18:54.034443 15051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "26837d8717cf4d27a32dcb57fb28a9d9"
format_stamp: "Formatted at 2026-09-28 02:18:54 on dist-test-slave-h37r"
server_key: "b400f2fe2cef26b1830ce03150245a74"
server_key_iv: "b3c717e02b35f8df0d47361f0633f26a"
server_key_version: "encryptionkey@0"
I20260928 02:18:54.040227 15051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:18:54.043874 15066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:54.045331 15051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:18:54.045526 15051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "26837d8717cf4d27a32dcb57fb28a9d9"
format_stamp: "Formatted at 2026-09-28 02:18:54 on dist-test-slave-h37r"
server_key: "b400f2fe2cef26b1830ce03150245a74"
server_key_iv: "b3c717e02b35f8df0d47361f0633f26a"
server_key_version: "encryptionkey@0"
I20260928 02:18:54.045785 15051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:54.061113 15051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:54.065646 15051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:54.065966 15051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:54.066903 15051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:54.068336 15051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:54.068439 15051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:54.068583 15051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:54.068673 15051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:54.101781 15051 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:36617
I20260928 02:18:54.101826 15178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:36617 every 8 connection(s)
I20260928 02:18:54.103098 15051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:18:54.113024 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 15051
I20260928 02:18:54.113276 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:18:54.114221 4659 external_mini_cluster.cc:1428] Setting key 9e2ad8d406c50c9ba926ca1b7a0e705e
Sep 28 02:18:54 dist-test-slave-h37r krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561934, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:18:54.125892 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:54.105770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:40193 (local address 127.4.140.254:39447)
0928 02:18:54.106234 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:54.106240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:54.107347 (+ 1107us) server_negotiation.cc:407] Connection header received
0928 02:18:54.109261 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:54.109266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:54.109337 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:54.109449 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:54.110946 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:54.111702 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:54.113506 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:54.114232 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:54.114394 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:54.118388 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:54.118398 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:54.118406 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:54.118456 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:54.122160 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:54.123075 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:54.123084 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:54.123090 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:54.123253 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:54.123750 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:54.123758 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:54.123765 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:54.124231 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:54.124402 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:54.125313 (+ 911us) server_negotiation.cc:299] Negotiation successful
0928 02:18:54.125532 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":218,"threads_started":1}
I20260928 02:18:54.126869 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:54.106428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:39447 (local address 127.4.140.193:40193)
0928 02:18:54.107192 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:54.107229 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:54.108971 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:54.109690 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:54.109698 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:54.110186 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:54.110768 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:54.110785 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:54.111849 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:54.111853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:54.113304 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:54.113316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:54.114492 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:54.114498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:54.114702 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:54.115485 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:54.115512 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:54.118138 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:54.122310 (+ 4172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:54.122321 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:54.122342 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:54.122917 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:54.123375 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:54.123381 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:54.123387 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:54.123619 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:54.124364 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:54.124374 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:54.124864 (+ 490us) client_negotiation.cc:769] Sending connection context
0928 02:18:54.125184 (+ 320us) client_negotiation.cc:240] Negotiation successful
0928 02:18:54.125741 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":330,"threads_started":1}
I20260928 02:18:54.128690 15179 heartbeater.cc:347] Connected to a master server at 127.4.140.254:39447
I20260928 02:18:54.129029 15179 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:54.129771 15179 heartbeater.cc:511] Master 127.4.140.254:39447 requested a full tablet report, sending...
I20260928 02:18:54.132195 14990 ts_manager.cc:194] Registered new tserver with Master: 26837d8717cf4d27a32dcb57fb28a9d9 (127.4.140.193:36617)
I20260928 02:18:54.134488 14990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:40193
WARNING: no policy specified for kudu/127.4.140.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
I20260928 02:18:54.169350 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:39447
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:54.494994 15185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:54.495368 15185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:54.495563 15185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:54.503221 15185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:54.503328 15185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:54.503438 15185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:18:54.513283 15185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:39447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:54.514535 15185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:54.515827 15185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:54.528630 15190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:54.528630 15191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:54.528697 15193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:54.529939 15185 server_base.cc:1034] running on GCE node
I20260928 02:18:54.530509 15185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:54.531416 15185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:54.532615 15185 hybrid_clock.cc:648] HybridClock initialized: now 1790561934532568 us; error 34 us; skew 500 ppm
I20260928 02:18:54.532928 15185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:54 dist-test-slave-h37r krb5kdc[14945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561934, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:54.536026 15185 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:18:54.537380 15185 webserver.cc:460] Webserver started at http://127.4.140.194:34493/ using document root <none> and password file <none>
I20260928 02:18:54.538030 15185 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:54.538170 15185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:54.538460 15185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:54.541371 15185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "5465bb5f17e2470d8d452392bd564b51"
format_stamp: "Formatted at 2026-09-28 02:18:54 on dist-test-slave-h37r"
server_key: "2900f8c24445f8f7f6097d863f5f7071"
server_key_iv: "3717bfb1f48dc719607455a3311d37c1"
server_key_version: "encryptionkey@0"
I20260928 02:18:54.542481 15185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "5465bb5f17e2470d8d452392bd564b51"
format_stamp: "Formatted at 2026-09-28 02:18:54 on dist-test-slave-h37r"
server_key: "2900f8c24445f8f7f6097d863f5f7071"
server_key_iv: "3717bfb1f48dc719607455a3311d37c1"
server_key_version: "encryptionkey@0"
I20260928 02:18:54.548300 15185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:18:54.551717 15200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:54.553171 15185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:18:54.553380 15185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "5465bb5f17e2470d8d452392bd564b51"
format_stamp: "Formatted at 2026-09-28 02:18:54 on dist-test-slave-h37r"
server_key: "2900f8c24445f8f7f6097d863f5f7071"
server_key_iv: "3717bfb1f48dc719607455a3311d37c1"
server_key_version: "encryptionkey@0"
I20260928 02:18:54.553639 15185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:54.574241 15185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:54.578433 15185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:54.578735 15185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:54.579663 15185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:54.581002 15185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:54.581075 15185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:54.581243 15185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:54.581292 15185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:54.614576 15185 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:33421
I20260928 02:18:54.614578 15312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:33421 every 8 connection(s)
I20260928 02:18:54.615978 15185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:18:54.621544 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 15185
I20260928 02:18:54.621735 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:18:54.622149 4659 external_mini_cluster.cc:1428] Setting key 032ad2e86e6fd2dddc2357ac15755a5b
Sep 28 02:18:54 dist-test-slave-h37r krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561934, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:18:54.635452 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:54.618571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:56535 (local address 127.4.140.254:39447)
0928 02:18:54.618752 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:54.618756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:54.619965 (+ 1209us) server_negotiation.cc:407] Connection header received
0928 02:18:54.621741 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:54.621746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:54.621862 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:54.621978 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:54.623573 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:54.624457 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:54.625974 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:54.626377 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:54.626507 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:54.629958 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:54.629963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:54.629966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:54.629997 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:54.632538 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:54.633212 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:54.633219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:54.633226 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:54.633357 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:54.633762 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:54.633768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:54.633772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:54.634018 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:54.634170 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:54.634872 (+ 702us) server_negotiation.cc:299] Negotiation successful
0928 02:18:54.635089 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:18:54.636499 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:54.619114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:39447 (local address 127.4.140.194:56535)
0928 02:18:54.619802 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:54.619839 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:54.621461 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:54.622209 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:54.622218 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:54.622811 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:54.623409 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:54.623426 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:54.624607 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:54.624618 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:54.625821 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:54.625832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:54.626594 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:54.626601 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:54.626829 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:54.627541 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:54.627588 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:54.629755 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:54.632672 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:54.632681 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:54.632697 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:54.633094 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:54.633464 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:54.633470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:54.633473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:54.633647 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:54.634142 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:54.634150 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:54.634497 (+ 347us) client_negotiation.cc:769] Sending connection context
0928 02:18:54.634777 (+ 280us) client_negotiation.cc:240] Negotiation successful
0928 02:18:54.635331 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":256,"threads_started":1}
I20260928 02:18:54.637997 15313 heartbeater.cc:347] Connected to a master server at 127.4.140.254:39447
I20260928 02:18:54.638319 15313 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:54.639050 15313 heartbeater.cc:511] Master 127.4.140.254:39447 requested a full tablet report, sending...
I20260928 02:18:54.640640 14990 ts_manager.cc:194] Registered new tserver with Master: 5465bb5f17e2470d8d452392bd564b51 (127.4.140.194:33421)
I20260928 02:18:54.641958 14990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:56535
WARNING: no policy specified for kudu/127.4.140.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
I20260928 02:18:54.674728 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:39447
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:18:55.003640 15318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:18:55.003964 15318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:18:55.004143 15318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:18:55.011539 15318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:18:55.011674 15318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:18:55.011883 15318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:18:55.022053 15318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:39447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:18:55.023409 15318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:18:55.024828 15318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:18:55.038300 15324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:55.038300 15326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:18:55.038412 15323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:18:55.039124 15318 server_base.cc:1034] running on GCE node
I20260928 02:18:55.039760 15318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:18:55.040777 15318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:18:55.042004 15318 hybrid_clock.cc:648] HybridClock initialized: now 1790561935041933 us; error 57 us; skew 500 ppm
I20260928 02:18:55.042366 15318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:18:55 dist-test-slave-h37r krb5kdc[14945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561935, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:18:55.045717 15318 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:18:55.047147 15318 webserver.cc:460] Webserver started at http://127.4.140.195:42467/ using document root <none> and password file <none>
I20260928 02:18:55.047839 15318 fs_manager.cc:359] Metadata directory not provided
I20260928 02:18:55.048005 15318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:18:55.048272 15318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:18:55.051329 15318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "b1f33d185efb4d719f429da95eb93503"
format_stamp: "Formatted at 2026-09-28 02:18:55 on dist-test-slave-h37r"
server_key: "a1dadb10d26262603ec1ae5471cb733d"
server_key_iv: "8e2083386d3a2e2abdfc5c3fd67e3200"
server_key_version: "encryptionkey@0"
I20260928 02:18:55.052139 15318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "b1f33d185efb4d719f429da95eb93503"
format_stamp: "Formatted at 2026-09-28 02:18:55 on dist-test-slave-h37r"
server_key: "a1dadb10d26262603ec1ae5471cb733d"
server_key_iv: "8e2083386d3a2e2abdfc5c3fd67e3200"
server_key_version: "encryptionkey@0"
I20260928 02:18:55.058108 15318 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 02:18:55.061877 15333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:55.063287 15318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:18:55.063458 15318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "b1f33d185efb4d719f429da95eb93503"
format_stamp: "Formatted at 2026-09-28 02:18:55 on dist-test-slave-h37r"
server_key: "a1dadb10d26262603ec1ae5471cb733d"
server_key_iv: "8e2083386d3a2e2abdfc5c3fd67e3200"
server_key_version: "encryptionkey@0"
I20260928 02:18:55.063673 15318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:18:55.090180 15318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:18:55.094316 15318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:18:55.094635 15318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:18:55.095856 15318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:18:55.097467 15318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:18:55.097600 15318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:55.097774 15318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:18:55.097859 15318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:18:55.133662 15318 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:39069
I20260928 02:18:55.133703 15445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:39069 every 8 connection(s)
I20260928 02:18:55.135003 15318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:18:55.137215 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 15318
I20260928 02:18:55.137416 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:18:55.138031 4659 external_mini_cluster.cc:1428] Setting key 8bf0f13af848484a14eb847e5be15917
I20260928 02:18:55.138418 15179 heartbeater.cc:503] Master 127.4.140.254:39447 was elected leader, sending a full tablet report...
Sep 28 02:18:55 dist-test-slave-h37r krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561935, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:18:55.156184 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:55.137791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:56723 (local address 127.4.140.254:39447)
0928 02:18:55.138362 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:55.138370 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:55.139624 (+ 1254us) server_negotiation.cc:407] Connection header received
0928 02:18:55.141840 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:55.141844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:55.141905 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:55.142005 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:55.143461 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.144300 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.145909 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.146277 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.146368 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:55.149857 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:55.149915 (+ 58us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:55.149920 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:55.149962 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:55.152841 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:55.153718 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:55.153725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:55.153735 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:55.153855 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:55.154316 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:55.154321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:55.154325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:55.154654 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:55.154808 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:55.155551 (+ 743us) server_negotiation.cc:299] Negotiation successful
0928 02:18:55.155759 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":259,"threads_started":1}
I20260928 02:18:55.157282 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:55.138523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:39447 (local address 127.4.140.195:56723)
0928 02:18:55.139456 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:55.139524 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:55.141586 (+ 2062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:55.142235 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:55.142243 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:55.142737 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:18:55.143311 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:55.143329 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.144461 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.144465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:55.145783 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:55.145799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.146454 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.146459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:55.146705 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:55.147347 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:18:55.147369 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:18:55.149650 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:18:55.152990 (+ 3340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:55.153000 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:55.153024 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:55.153596 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:55.153976 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:18:55.153981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:18:55.153985 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:18:55.154186 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:18:55.154785 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:18:55.154793 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:18:55.155134 (+ 341us) client_negotiation.cc:769] Sending connection context
0928 02:18:55.155446 (+ 312us) client_negotiation.cc:240] Negotiation successful
0928 02:18:55.156034 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":364,"threads_started":1}
I20260928 02:18:55.158696 15446 heartbeater.cc:347] Connected to a master server at 127.4.140.254:39447
I20260928 02:18:55.159016 15446 heartbeater.cc:464] Registering TS with master...
I20260928 02:18:55.159741 15446 heartbeater.cc:511] Master 127.4.140.254:39447 requested a full tablet report, sending...
I20260928 02:18:55.161262 14989 ts_manager.cc:194] Registered new tserver with Master: b1f33d185efb4d719f429da95eb93503 (127.4.140.195:39069)
I20260928 02:18:55.162194 14989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:56723
I20260928 02:18:55.166090 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:18:55.181958 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:55.170459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51980 (local address 127.4.140.254:39447)
0928 02:18:55.170694 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:55.170701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:55.171001 (+ 300us) server_negotiation.cc:407] Connection header received
0928 02:18:55.171114 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:55.171117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:55.171176 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:55.171252 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:55.172677 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.173364 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.174589 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.174974 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.175073 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:55.177334 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:55.177338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:55.177342 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:55.177369 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:55.179653 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:55.180280 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:55.180284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:55.180286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:55.180390 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:55.180710 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:55.180713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:55.180715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:55.181000 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:55.181183 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:55.181461 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 02:18:55.181628 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:18:55.185947 14989 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:18:55.192833 14989 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:18:55.226926 15461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:55.205652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:33421 (local address 127.0.0.1:46518)
0928 02:18:55.217934 (+ 12282us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:55.217968 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:55.218125 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:55.219088 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:55.219100 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:55.219145 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:55.219733 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:55.219750 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.221661 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.221667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:55.224363 (+ 2696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:55.224375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.225775 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.225781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:55.226074 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:55.226145 (+ 71us) client_negotiation.cc:769] Sending connection context
0928 02:18:55.226270 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 02:18:55.226482 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12106,"mutex_wait_us":60,"thread_start_us":11883,"threads_started":1}
I20260928 02:18:55.227154 15462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:55.206585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46518 (local address 127.4.140.194:33421)
0928 02:18:55.218244 (+ 11659us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:55.218256 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:55.218296 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:18:55.218423 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:55.218435 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:55.218781 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:55.219046 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:55.220020 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.221494 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.224524 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.225568 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.225717 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:55.225920 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:55.226683 (+ 763us) server_negotiation.cc:299] Negotiation successful
0928 02:18:55.226858 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11426,"thread_start_us":11644,"threads_started":1}
I20260928 02:18:55.227782 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:55.204924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:36617 (local address 127.0.0.1:57378)
0928 02:18:55.205856 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:55.205921 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:55.206132 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:55.221281 (+ 15149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:55.221288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:55.221314 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:55.221887 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:55.221899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.223225 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.223233 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:55.225239 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:55.225252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.226942 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.226950 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:55.227109 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:55.227140 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:18:55.227211 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 02:18:55.227336 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":4,"thread_start_us":904,"threads_started":1}
I20260928 02:18:55.228916 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:55.205213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.4.140.193:36617)
0928 02:18:55.220770 (+ 15557us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:55.220778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:55.220805 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:18:55.220879 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:55.220885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:55.221093 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:55.221778 (+ 685us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:55.222066 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.223093 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.225404 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.226699 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.228131 (+ 1432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:55.228376 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:55.228442 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 02:18:55.228596 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15392,"thread_start_us":15257,"threads_started":1}
I20260928 02:18:55.229768 15463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:55.206753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:39069 (local address 127.0.0.1:42024)
0928 02:18:55.219778 (+ 13025us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:55.219811 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:55.220577 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:55.221553 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:55.221559 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:55.221590 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:55.222123 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:55.222138 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.224097 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.224103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:55.226132 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:55.226143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.229003 (+ 2860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.229010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:55.229160 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:55.229196 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:18:55.229289 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 02:18:55.229419 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12831,"mutex_wait_us":42,"thread_start_us":12688,"threads_started":1}
I20260928 02:18:55.230096 15464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:55.207415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42024 (local address 127.4.140.195:39069)
0928 02:18:55.220436 (+ 13021us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:55.220447 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:55.220485 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:18:55.220781 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:55.220791 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:55.221037 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:55.221234 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:55.222324 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.223948 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.226283 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.228798 (+ 2515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.228931 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:55.229151 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:55.229422 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 02:18:55.229569 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12841,"thread_start_us":12668,"threads_started":1}
I20260928 02:18:55.231195 15248 tablet_service.cc:1467] Processing CreateTablet for tablet e6763e3270604dcd8b4822505fb2c7f4 (DEFAULT_TABLE table=test-table [id=b52a0f6da8094b8889e9780080ae8022]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:55.231980 15114 tablet_service.cc:1467] Processing CreateTablet for tablet e6763e3270604dcd8b4822505fb2c7f4 (DEFAULT_TABLE table=test-table [id=b52a0f6da8094b8889e9780080ae8022]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:55.232623 15248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6763e3270604dcd8b4822505fb2c7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:55.233239 15114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6763e3270604dcd8b4822505fb2c7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:55.235200 15381 tablet_service.cc:1467] Processing CreateTablet for tablet e6763e3270604dcd8b4822505fb2c7f4 (DEFAULT_TABLE table=test-table [id=b52a0f6da8094b8889e9780080ae8022]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:18:55.236553 15381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6763e3270604dcd8b4822505fb2c7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:18:55.245007 15465 tablet_bootstrap.cc:492] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9: Bootstrap starting.
I20260928 02:18:55.247308 15466 tablet_bootstrap.cc:492] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51: Bootstrap starting.
I20260928 02:18:55.248878 15465 tablet_bootstrap.cc:654] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:55.250334 15467 tablet_bootstrap.cc:492] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503: Bootstrap starting.
I20260928 02:18:55.250355 15465 log.cc:824] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:55.250947 15466 tablet_bootstrap.cc:654] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:55.252521 15466 log.cc:824] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:55.253166 15467 tablet_bootstrap.cc:654] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503: Neither blocks nor log segments found. Creating new log.
I20260928 02:18:55.254382 15467 log.cc:824] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503: Log is configured to *not* fsync() on all Append() calls
I20260928 02:18:55.255852 15466 tablet_bootstrap.cc:492] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51: No bootstrap required, opened a new log
I20260928 02:18:55.256177 15466 ts_tablet_manager.cc:1397] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260928 02:18:55.256981 15465 tablet_bootstrap.cc:492] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9: No bootstrap required, opened a new log
I20260928 02:18:55.257243 15465 ts_tablet_manager.cc:1397] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260928 02:18:55.257553 15467 tablet_bootstrap.cc:492] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503: No bootstrap required, opened a new log
I20260928 02:18:55.257844 15467 ts_tablet_manager.cc:1397] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 02:18:55.265044 15465 raft_consensus.cc:348] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26837d8717cf4d27a32dcb57fb28a9d9" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36617 } } peers { permanent_uuid: "b1f33d185efb4d719f429da95eb93503" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 39069 } } peers { permanent_uuid: "5465bb5f17e2470d8d452392bd564b51" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 33421 } }
I20260928 02:18:55.265511 15465 raft_consensus.cc:374] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:55.265604 15465 raft_consensus.cc:729] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26837d8717cf4d27a32dcb57fb28a9d9, State: Initialized, Role: FOLLOWER
I20260928 02:18:55.266222 15465 consensus_queue.cc:260] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26837d8717cf4d27a32dcb57fb28a9d9" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36617 } } peers { permanent_uuid: "b1f33d185efb4d719f429da95eb93503" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 39069 } } peers { permanent_uuid: "5465bb5f17e2470d8d452392bd564b51" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 33421 } }
I20260928 02:18:55.266963 15466 raft_consensus.cc:348] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26837d8717cf4d27a32dcb57fb28a9d9" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36617 } } peers { permanent_uuid: "b1f33d185efb4d719f429da95eb93503" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 39069 } } peers { permanent_uuid: "5465bb5f17e2470d8d452392bd564b51" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 33421 } }
I20260928 02:18:55.267408 15466 raft_consensus.cc:374] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:55.267503 15466 raft_consensus.cc:729] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5465bb5f17e2470d8d452392bd564b51, State: Initialized, Role: FOLLOWER
I20260928 02:18:55.268009 15465 ts_tablet_manager.cc:1428] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260928 02:18:55.268306 15466 consensus_queue.cc:260] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26837d8717cf4d27a32dcb57fb28a9d9" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36617 } } peers { permanent_uuid: "b1f33d185efb4d719f429da95eb93503" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 39069 } } peers { permanent_uuid: "5465bb5f17e2470d8d452392bd564b51" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 33421 } }
I20260928 02:18:55.269567 15313 heartbeater.cc:503] Master 127.4.140.254:39447 was elected leader, sending a full tablet report...
I20260928 02:18:55.270241 15466 ts_tablet_manager.cc:1428] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260928 02:18:55.270023 15467 raft_consensus.cc:348] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26837d8717cf4d27a32dcb57fb28a9d9" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36617 } } peers { permanent_uuid: "b1f33d185efb4d719f429da95eb93503" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 39069 } } peers { permanent_uuid: "5465bb5f17e2470d8d452392bd564b51" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 33421 } }
I20260928 02:18:55.270700 15467 raft_consensus.cc:374] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:18:55.270805 15467 raft_consensus.cc:729] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f33d185efb4d719f429da95eb93503, State: Initialized, Role: FOLLOWER
I20260928 02:18:55.271370 15467 consensus_queue.cc:260] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26837d8717cf4d27a32dcb57fb28a9d9" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36617 } } peers { permanent_uuid: "b1f33d185efb4d719f429da95eb93503" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 39069 } } peers { permanent_uuid: "5465bb5f17e2470d8d452392bd564b51" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 33421 } }
I20260928 02:18:55.272641 15446 heartbeater.cc:503] Master 127.4.140.254:39447 was elected leader, sending a full tablet report...
I20260928 02:18:55.273223 15467 ts_tablet_manager.cc:1428] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503: Time spent starting tablet: real 0.015s user 0.007s sys 0.006s
W20260928 02:18:55.355577 15180 tablet.cc:2369] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:18:55.368032 15314 tablet.cc:2369] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:18:55.385908 15471 raft_consensus.cc:482] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:18:55.386178 15471 raft_consensus.cc:504] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26837d8717cf4d27a32dcb57fb28a9d9" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36617 } } peers { permanent_uuid: "b1f33d185efb4d719f429da95eb93503" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 39069 } } peers { permanent_uuid: "5465bb5f17e2470d8d452392bd564b51" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 33421 } }
W20260928 02:18:55.386714 15447 tablet.cc:2369] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:18:55.387866 15471 leader_election.cc:288] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1f33d185efb4d719f429da95eb93503 (127.4.140.195:39069), 5465bb5f17e2470d8d452392bd564b51 (127.4.140.194:33421)
I20260928 02:18:55.394588 15462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:55.388452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:41757 (local address 127.4.140.194:33421)
0928 02:18:55.388710 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:55.388718 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:55.389423 (+ 705us) server_negotiation.cc:407] Connection header received
0928 02:18:55.389586 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:55.389590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:55.389646 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:55.389740 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:55.390360 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.391040 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.392934 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.393688 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.393866 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:55.393940 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:55.394217 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 02:18:55.394339 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:18:55.394657 15464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:55.388330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:53217 (local address 127.4.140.195:39069)
0928 02:18:55.388750 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:55.388760 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:55.389304 (+ 544us) server_negotiation.cc:407] Connection header received
0928 02:18:55.389547 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:55.389551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:55.389606 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:55.389706 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:18:55.390301 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.391048 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.392965 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.393720 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.393894 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:55.393971 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:55.394307 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 02:18:55.394424 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260928 02:18:55.395028 15475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:55.388176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:33421 (local address 127.4.140.193:41757)
0928 02:18:55.389311 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:55.389329 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:55.389441 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:55.389764 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:55.389768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:55.389790 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:55.390214 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:55.390222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.391174 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.391182 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:55.392778 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:55.392789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.393904 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.393911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:55.394067 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:55.394089 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:18:55.394193 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:18:55.394314 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"mutex_wait_us":142,"thread_start_us":728,"threads_started":1}
I20260928 02:18:55.395431 15474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:55.388176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:39069 (local address 127.4.140.193:53217)
0928 02:18:55.389188 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0928 02:18:55.389203 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:18:55.389397 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:18:55.389719 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:18:55.389723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:18:55.389746 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:18:55.390152 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:55.390164 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.391173 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.391181 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:55.392814 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:18:55.392825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.393951 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.393958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:18:55.394105 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:55.394126 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:18:55.394845 (+ 719us) client_negotiation.cc:240] Negotiation successful
0928 02:18:55.394977 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":44,"thread_start_us":654,"threads_started":1}
I20260928 02:18:55.396091 15268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6763e3270604dcd8b4822505fb2c7f4" candidate_uuid: "26837d8717cf4d27a32dcb57fb28a9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5465bb5f17e2470d8d452392bd564b51" is_pre_election: true
I20260928 02:18:55.396572 15268 raft_consensus.cc:2454] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26837d8717cf4d27a32dcb57fb28a9d9 in term 0.
I20260928 02:18:55.396509 15401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6763e3270604dcd8b4822505fb2c7f4" candidate_uuid: "26837d8717cf4d27a32dcb57fb28a9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1f33d185efb4d719f429da95eb93503" is_pre_election: true
I20260928 02:18:55.396931 15401 raft_consensus.cc:2454] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26837d8717cf4d27a32dcb57fb28a9d9 in term 0.
I20260928 02:18:55.397207 15067 leader_election.cc:302] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26837d8717cf4d27a32dcb57fb28a9d9, 5465bb5f17e2470d8d452392bd564b51; no voters:
I20260928 02:18:55.397579 15471 raft_consensus.cc:2781] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:18:55.397727 15471 raft_consensus.cc:482] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:18:55.397835 15471 raft_consensus.cc:3037] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:55.399889 15471 raft_consensus.cc:504] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26837d8717cf4d27a32dcb57fb28a9d9" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36617 } } peers { permanent_uuid: "b1f33d185efb4d719f429da95eb93503" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 39069 } } peers { permanent_uuid: "5465bb5f17e2470d8d452392bd564b51" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 33421 } }
I20260928 02:18:55.400484 15471 leader_election.cc:288] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [CANDIDATE]: Term 1 election: Requested vote from peers b1f33d185efb4d719f429da95eb93503 (127.4.140.195:39069), 5465bb5f17e2470d8d452392bd564b51 (127.4.140.194:33421)
I20260928 02:18:55.400928 15268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6763e3270604dcd8b4822505fb2c7f4" candidate_uuid: "26837d8717cf4d27a32dcb57fb28a9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5465bb5f17e2470d8d452392bd564b51"
I20260928 02:18:55.400947 15401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6763e3270604dcd8b4822505fb2c7f4" candidate_uuid: "26837d8717cf4d27a32dcb57fb28a9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1f33d185efb4d719f429da95eb93503"
I20260928 02:18:55.401103 15268 raft_consensus.cc:3037] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:55.401127 15401 raft_consensus.cc:3037] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:18:55.403172 15401 raft_consensus.cc:2454] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26837d8717cf4d27a32dcb57fb28a9d9 in term 1.
I20260928 02:18:55.403218 15268 raft_consensus.cc:2454] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26837d8717cf4d27a32dcb57fb28a9d9 in term 1.
I20260928 02:18:55.403560 15068 leader_election.cc:302] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26837d8717cf4d27a32dcb57fb28a9d9, b1f33d185efb4d719f429da95eb93503; no voters:
I20260928 02:18:55.403944 15471 raft_consensus.cc:2781] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:18:55.404392 15471 raft_consensus.cc:686] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [term 1 LEADER]: Becoming Leader. State: Replica: 26837d8717cf4d27a32dcb57fb28a9d9, State: Running, Role: LEADER
I20260928 02:18:55.404898 15471 consensus_queue.cc:237] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26837d8717cf4d27a32dcb57fb28a9d9" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36617 } } peers { permanent_uuid: "b1f33d185efb4d719f429da95eb93503" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 39069 } } peers { permanent_uuid: "5465bb5f17e2470d8d452392bd564b51" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 33421 } }
I20260928 02:18:55.409118 14989 catalog_manager.cc:5495] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26837d8717cf4d27a32dcb57fb28a9d9 (127.4.140.193). New cstate: current_term: 1 leader_uuid: "26837d8717cf4d27a32dcb57fb28a9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26837d8717cf4d27a32dcb57fb28a9d9" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1f33d185efb4d719f429da95eb93503" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5465bb5f17e2470d8d452392bd564b51" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 33421 } health_report { overall_health: UNKNOWN } } }
I20260928 02:18:55.453727 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:55.448397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56874 (local address 127.4.140.193:36617)
0928 02:18:55.448629 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:55.448637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:55.448669 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:18:55.448755 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:55.448759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:55.448847 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:55.449002 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:18:55.449389 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.450144 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.451520 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.451986 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.452115 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:55.452591 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:18:55.453139 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:18:55.453306 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:55.453355 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 02:18:55.453432 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 02:18:55.462581 15401 raft_consensus.cc:1261] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503 [term 1 FOLLOWER]: Refusing update from remote peer 26837d8717cf4d27a32dcb57fb28a9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:18:55.462584 15268 raft_consensus.cc:1261] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51 [term 1 FOLLOWER]: Refusing update from remote peer 26837d8717cf4d27a32dcb57fb28a9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:18:55.463559 15476 consensus_queue.cc:1035] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5465bb5f17e2470d8d452392bd564b51" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 33421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:18:55.463937 15471 consensus_queue.cc:1035] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1f33d185efb4d719f429da95eb93503" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:18:55.475468 15479 mvcc.cc:205] Tried to move back new op lower bound from 7334141687643189248 to 7334141687422091264. Current Snapshot: MvccSnapshot[applied={T|T < 7334141687643189248}]
I20260928 02:18:55.477489 15481 mvcc.cc:205] Tried to move back new op lower bound from 7334141687643189248 to 7334141687422091264. Current Snapshot: MvccSnapshot[applied={T|T < 7334141687643189248}]
I20260928 02:18:55.479491 15480 mvcc.cc:205] Tried to move back new op lower bound from 7334141687643189248 to 7334141687422091264. Current Snapshot: MvccSnapshot[applied={T|T < 7334141687643189248}]
I20260928 02:18:55.487501 14989 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:18:55.487759 14989 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:18:55.491042 14989 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1dbe2c94f09b4cd1bb396363c9ed0bd3: Sending DeleteTablet for 3 replicas of tablet e6763e3270604dcd8b4822505fb2c7f4
I20260928 02:18:55.492077 15114 tablet_service.cc:1514] Processing DeleteTablet for tablet e6763e3270604dcd8b4822505fb2c7f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:55 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:57378
I20260928 02:18:55.492231 15381 tablet_service.cc:1514] Processing DeleteTablet for tablet e6763e3270604dcd8b4822505fb2c7f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:55 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:42024
I20260928 02:18:55.492304 15248 tablet_service.cc:1514] Processing DeleteTablet for tablet e6763e3270604dcd8b4822505fb2c7f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:18:55 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:46518
I20260928 02:18:55.493790 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 15051
I20260928 02:18:55.504467 15490 tablet_replica.cc:330] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51: stopping tablet replica
I20260928 02:18:55.504668 15489 tablet_replica.cc:330] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503: stopping tablet replica
I20260928 02:18:55.504951 15490 raft_consensus.cc:2229] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:18:55.505007 15489 raft_consensus.cc:2229] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:18:55.505470 15490 raft_consensus.cc:2258] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:55.505470 15489 raft_consensus.cc:2258] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:55.505769 15488 tablet_replica.cc:330] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9: stopping tablet replica
I20260928 02:18:55.506179 15488 raft_consensus.cc:2229] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:18:55.506867 15488 raft_consensus.cc:2258] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:18:55.507675 15489 ts_tablet_manager.cc:1905] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:55.507669 15490 ts_tablet_manager.cc:1905] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:55.508967 15488 ts_tablet_manager.cc:1905] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:18:55.514659 15490 ts_tablet_manager.cc:1918] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:18:55.514815 15490 log.cc:1196] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/e6763e3270604dcd8b4822505fb2c7f4
I20260928 02:18:55.515317 15490 ts_tablet_manager.cc:1939] T e6763e3270604dcd8b4822505fb2c7f4 P 5465bb5f17e2470d8d452392bd564b51: Deleting consensus metadata
Sep 28 02:18:55 dist-test-slave-h37r krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:18:55.516994 15488 ts_tablet_manager.cc:1918] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:18:55.517069 14974 catalog_manager.cc:4841] TS 5465bb5f17e2470d8d452392bd564b51 (127.4.140.194:33421): tablet e6763e3270604dcd8b4822505fb2c7f4 (table test-table [id=b52a0f6da8094b8889e9780080ae8022]) successfully deleted
I20260928 02:18:55.517293 15488 log.cc:1196] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/e6763e3270604dcd8b4822505fb2c7f4
I20260928 02:18:55.518270 15488 ts_tablet_manager.cc:1939] T e6763e3270604dcd8b4822505fb2c7f4 P 26837d8717cf4d27a32dcb57fb28a9d9: Deleting consensus metadata
I20260928 02:18:55.520386 14975 catalog_manager.cc:4841] TS 26837d8717cf4d27a32dcb57fb28a9d9 (127.4.140.193:36617): tablet e6763e3270604dcd8b4822505fb2c7f4 (table test-table [id=b52a0f6da8094b8889e9780080ae8022]) successfully deleted
I20260928 02:18:55.524295 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:55.495296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.4.140.193:36617)
0928 02:18:55.495525 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:55.495535 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:55.505018 (+ 9483us) server_negotiation.cc:407] Connection header received
0928 02:18:55.505305 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:55.505313 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:55.505407 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:55.505527 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:55.507378 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.508490 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.510280 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.510968 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.511076 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:55.516563 (+ 5487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:55.516578 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:55.516597 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:55.516665 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:55.520802 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:55.521667 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:55.521676 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:55.521680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:55.521848 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:55.522322 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:55.522327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:55.522330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:55.522765 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:55.523039 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:55.523642 (+ 603us) server_negotiation.cc:299] Negotiation successful
0928 02:18:55.523946 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:18:55.524284 15489 ts_tablet_manager.cc:1918] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:18:55.524426 15489 log.cc:1196] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/e6763e3270604dcd8b4822505fb2c7f4
I20260928 02:18:55.525279 15489 ts_tablet_manager.cc:1939] T e6763e3270604dcd8b4822505fb2c7f4 P b1f33d185efb4d719f429da95eb93503: Deleting consensus metadata
I20260928 02:18:55.525735 15174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:55.526868 14975 catalog_manager.cc:4841] TS b1f33d185efb4d719f429da95eb93503 (127.4.140.195:39069): tablet e6763e3270604dcd8b4822505fb2c7f4 (table test-table [id=b52a0f6da8094b8889e9780080ae8022]) successfully deleted
I20260928 02:18:55.624511 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 15051
I20260928 02:18:55.643958 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 15185
Sep 28 02:18:55 dist-test-slave-h37r krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.194@KRBTEST.COM
I20260928 02:18:55.658476 15462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:55.644813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40512 (local address 127.4.140.194:33421)
0928 02:18:55.644989 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:55.644994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:55.645012 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:18:55.645072 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:55.645076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:55.645146 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:55.645237 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:55.646465 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.647241 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.648723 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.649113 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.649209 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:55.653188 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:55.653200 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:55.653219 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:55.653250 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:55.656003 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:55.656592 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:55.656598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:55.656600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:55.656724 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:55.657066 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:55.657069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:55.657071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:55.657437 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:55.657651 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:55.657979 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 02:18:55.658178 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:18:55.659137 15308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:55.748561 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 15185
I20260928 02:18:55.766050 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 15318
Sep 28 02:18:55 dist-test-slave-h37r krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.195@KRBTEST.COM
I20260928 02:18:55.780645 15464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:18:55.766793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48822 (local address 127.4.140.195:39069)
0928 02:18:55.766959 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:18:55.766963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:18:55.766982 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:18:55.767147 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:18:55.767151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:18:55.767253 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:18:55.767350 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:18:55.768684 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.769457 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.770639 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:18:55.771193 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:18:55.771280 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:18:55.775594 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:18:55.775608 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:18:55.775627 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:18:55.775671 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:18:55.778330 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:55.778949 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:55.778956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:55.778959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:55.779095 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:18:55.779408 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:18:55.779411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:18:55.779413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:18:55.779748 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:18:55.780124 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0928 02:18:55.780244 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 02:18:55.780395 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:18:55.781288 15441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:55.869154 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 15318
I20260928 02:18:55.886134 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 14958
I20260928 02:18:55.887436 15020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:18:55.995841 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 14958
2026-09-28T02:18:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4263 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:18:56 dist-test-slave-h37r krb5kdc[15498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:18:56 dist-test-slave-h37r krb5kdc[15498](info): set up 2 sockets
Sep 28 02:18:56 dist-test-slave-h37r krb5kdc[15498](info): commencing operation
krb5kdc: starting...
W20260928 02:18:57.104887 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:18:57 dist-test-slave-h37r krb5kdc[15498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:18:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:18:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.140.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.140.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.140.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/rangeradmin_127.4.140.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.140.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/rangeradmin_127.4.140.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.140.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.140.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.140.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/rangerlookup_127.4.140.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.140.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/rangerlookup_127.4.140.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.140.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.140.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.140.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/HTTP_127.4.140.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.140.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/HTTP_127.4.140.212@KRBTEST.COM.keytab.
I20260928 02:18:57.439278 4659 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskx4rqyh/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/postgres -l logfile start
2026-09-28 02:19:00.029 UTC [15528] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:19:00.030 UTC [15528] LOG: listening on IPv4 address "127.4.140.212", port 35207
2026-09-28 02:19:00.033 UTC [15528] LOG: listening on Unix socket "/tmp/.s.PGSQL.35207"
2026-09-28 02:19:00.055 UTC [15531] LOG: database system was shut down at 2026-09-28 02:18:58 UTC
2026-09-28 02:19:00.059 UTC [15528] LOG: database system is ready to accept connections
I20260928 02:19:01.056440 4659 mini_postgres.cc:96] Postgres bound to 35207
2026-09-28 02:19:01.073 UTC [15539] FATAL: database "slave" does not exist
127.4.140.212:35207 - accepting connections
I20260928 02:19:01.074618 4659 mini_ranger.cc:161] Starting Ranger...
I20260928 02:19:01.101464 4659 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:19:01.623438 4659 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:19:01.623600 4659 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ranger-admin
2026-09-28 02:19:01,905 [I] DB FLAVOR :POSTGRES
2026-09-28 02:19:01,905 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:19:01,906 [I] Checking connection..
2026-09-28 02:19:01,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:19:02,174 [I] Checking connection passed.
2026-09-28 02:19:02,175 [I] --------- Verifying version history table ---------
2026-09-28 02:19:02,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:19:02,429 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:19:02,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:19:02,669 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:19:02,669 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:19:02,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:19:02.914 UTC [15603] WARNING: there is no transaction in progress
2026-09-28 02:19:02,920 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:19:02,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:19:03,156 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:19:03,156 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:19:03,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:19:03,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:19:03,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-h37r.c.gcp-upstream.internal','N') ;"
2026-09-28 02:19:03,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:19:04,139 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:19:04,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:19:04,412 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:19:04,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:19:04,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:19:04,910 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:19:04,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:19:05.857 UTC [15711] WARNING: there is no transaction in progress
2026-09-28 02:19:05.874 UTC [15711] WARNING: there is no transaction in progress
2026-09-28 02:19:05.891 UTC [15711] WARNING: there is no transaction in progress
2026-09-28 02:19:05.907 UTC [15711] WARNING: there is no transaction in progress
2026-09-28 02:19:05.927 UTC [15711] WARNING: there is no transaction in progress
2026-09-28 02:19:05.946 UTC [15711] WARNING: there is no transaction in progress
2026-09-28 02:19:06.004 UTC [15711] WARNING: there is no transaction in progress
2026-09-28 02:19:06.011 UTC [15711] WARNING: there is no transaction in progress
2026-09-28 02:19:06.027 UTC [15711] WARNING: there is no transaction in progress
2026-09-28 02:19:06.037 UTC [15711] WARNING: there is no transaction in progress
2026-09-28 02:19:06.048 UTC [15711] WARNING: there is no transaction in progress
2026-09-28 02:19:06.061 UTC [15711] WARNING: there is no transaction in progress
2026-09-28 02:19:06.071 UTC [15711] WARNING: there is no transaction in progress
2026-09-28 02:19:06.077 UTC [15711] WARNING: there is no transaction in progress
2026-09-28 02:19:06.085 UTC [15711] WARNING: there is no transaction in progress
2026-09-28 02:19:06.092 UTC [15711] WARNING: there is no transaction in progress
2026-09-28 02:19:06,742 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:19:06,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:19:06,966 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:19:06,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:19:07,218 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:19:07,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:19:07,483 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:19:07,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:19:07,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:19:08,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-h37r.c.gcp-upstream.internal';"
2026-09-28 02:19:08,269 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:19:08,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:19:08,524 [I] DB_PATCHES have already been applied
I20260928 02:19:08.530313 4659 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ranger-admin:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskx4rqyh/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:19:08.530479 4659 mini_ranger.cc:193] Using host: 127.4.140.212
I20260928 02:19:08.546100 4659 mini_ranger.cc:227] Ranger admin URL: http://127.4.140.212:34105
Sep 28, 2026 2:19:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:19:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:19:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:19:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:19:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:19:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:19:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:19:09.695235 4659 mini_ranger.cc:160] Time spent starting Ranger: real 8.621s user 0.003s sys 0.039s
Sep 28, 2026 2:19:29 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:19:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:19:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:19:29 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:19:29 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:19:30 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 02:19:30.694031 4659 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.140.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.140.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.140.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/rangerkms_127.4.140.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.140.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/rangerkms_127.4.140.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.140.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.140.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.140.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/HTTP_127.4.140.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.140.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/HTTP_127.4.140.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 02:19:30 dist-test-slave-h37r krb5kdc[15498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561970, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 02:19:30.863933 4659 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 02:19:30.884635 4659 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 02:19:31.824875 4659 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 02:19:31.825007 4659 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ranger-kms
I20260928 02:19:31.825155 4659 mini_ranger_kms.cc:227] Using postgres at 127.4.140.212:35207
2026-09-28 02:19:32,256 [I] DB FLAVOR :POSTGRES
2026-09-28 02:19:32,256 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:19:32,256 [I] Checking connection
2026-09-28 02:19:32,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 02:19:32,500 [I] connection success
2026-09-28 02:19:32,500 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 02:19:32,500 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 02:19:32,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 02:19:32,749 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 02:19:32,749 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:19:32,750 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 02:19:32,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:35207/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 02:19:33,002 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 02:19:33.457741 4659 mini_ranger_kms.cc:310] Created kms service
I20260928 02:19:33.637773 4659 mini_ranger_kms.cc:326] Created kudu user
I20260928 02:19:33.687150 4659 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 02:19:33 dist-test-slave-h37r krb5kdc[15498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561970, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
I20260928 02:19:34.116216 4659 mini_ranger_kms.cc:384] Added ranger policy
I20260928 02:19:34.116530 4659 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ranger-kms:/tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 02:19:34.116598 4659 mini_ranger_kms.cc:240] Using host: 127.4.140.212
I20260928 02:19:34.137830 4659 mini_ranger_kms.cc:276] Ranger KMS PID: 15891
I20260928 02:19:34.138038 4659 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.140.212:39781
Sep 28, 2026 2:19:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:19:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:19:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:19:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:19:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:19:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 02:19:36 dist-test-slave-h37r krb5kdc[15498](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790561976, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.140.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:19:36.563278 4659 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.699s user 0.125s sys 0.327s
I20260928 02:19:36.563548 4659 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 02:19:36.563769 4659 mini_ranger_kms.cc:401] 127.4.140.212:39781/kms/v1/keys
Sep 28 02:19:36 dist-test-slave-h37r krb5kdc[15498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561976, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.140.212@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
Sep 28 02:19:37 dist-test-slave-h37r krb5kdc[15498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561976, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.140.212@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
Sep 28 02:19:39 dist-test-slave-h37r krb5kdc[15498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:19:39.443841 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:36211
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:36211
--ranger_config_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.140.212:39781/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:19:39.780085 15942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:19:39.780382 15942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:19:39.780527 15942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:19:39.788028 15942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:19:39.788136 15942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:19:39.788224 15942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:19:39.788317 15942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:19:39.788406 15942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:19:39.798978 15942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.140.212:39781/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:36211
--ranger_config_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:36211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:19:39.800436 15942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:19:39.801785 15942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:19:39.815876 15947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:19:39.815855 15948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:19:39.816056 15950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:19:39.816807 15942 server_base.cc:1034] running on GCE node
I20260928 02:19:39.817636 15942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:19:39.819311 15942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:19:39.820529 15942 hybrid_clock.cc:648] HybridClock initialized: now 1790561979820474 us; error 51 us; skew 500 ppm
I20260928 02:19:39.820899 15942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:19:39 dist-test-slave-h37r krb5kdc[15498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561979, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:19:39.828452 15942 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:19:39.830228 15942 webserver.cc:460] Webserver started at http://127.4.140.254:45363/ using document root <none> and password file <none>
I20260928 02:19:39.830915 15942 fs_manager.cc:359] Metadata directory not provided
I20260928 02:19:39.831058 15942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:19:39.831393 15942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:19:39 dist-test-slave-h37r krb5kdc[15498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561979, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
I20260928 02:19:39.882009 15942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "4263109436924d6f914469ced887072b"
format_stamp: "Formatted at 2026-09-28 02:19:39 on dist-test-slave-h37r"
server_key: "f4a430b5558dd7b484196c9d189e4274"
server_key_iv: "a072c3537633c27f01c0126d39988ac1"
server_key_version: "kuduclusterkey@0"
I20260928 02:19:39.882863 15942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "4263109436924d6f914469ced887072b"
format_stamp: "Formatted at 2026-09-28 02:19:39 on dist-test-slave-h37r"
server_key: "f4a430b5558dd7b484196c9d189e4274"
server_key_iv: "a072c3537633c27f01c0126d39988ac1"
server_key_version: "kuduclusterkey@0"
I20260928 02:19:39.888602 15942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:19:39.905386 15957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:19:39.907008 15942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:19:39.907197 15942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "4263109436924d6f914469ced887072b"
format_stamp: "Formatted at 2026-09-28 02:19:39 on dist-test-slave-h37r"
server_key: "f4a430b5558dd7b484196c9d189e4274"
server_key_iv: "a072c3537633c27f01c0126d39988ac1"
server_key_version: "kuduclusterkey@0"
I20260928 02:19:39.907421 15942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:19:39.924961 15942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:19:39.928977 15942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:19:39.929288 15942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:19:39.949160 15942 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:36211
I20260928 02:19:39.949184 16008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:36211 every 8 connection(s)
I20260928 02:19:39.950562 15942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:19:39.950907 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 15942
I20260928 02:19:39.951112 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:19:39.951948 4659 external_mini_cluster.cc:1428] Setting key de8e1a9f7fa7fd9eae3346b732b4685e
I20260928 02:19:39.956453 16009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:19:39.972924 16009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b: Bootstrap starting.
Sep 28 02:19:39 dist-test-slave-h37r krb5kdc[15498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:19:39.976858 16009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b: Neither blocks nor log segments found. Creating new log.
I20260928 02:19:39.978315 16009 log.cc:824] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:19:39.981678 16009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b: No bootstrap required, opened a new log
I20260928 02:19:39.984143 16012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:39.954516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46292 (local address 127.4.140.254:36211)
0928 02:19:39.962898 (+ 8382us) server_negotiation.cc:206] Beginning negotiation
0928 02:19:39.962919 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 02:19:39.962991 (+ 72us) server_negotiation.cc:407] Connection header received
0928 02:19:39.965141 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:19:39.965167 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:19:39.965650 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:19:39.966050 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:19:39.967631 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:39.969062 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:39.970902 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:39.971550 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:39.971677 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:39.976037 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:19:39.976048 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:19:39.976068 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:19:39.976114 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:19:39.979692 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:19:39.980394 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:19:39.980402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:19:39.980411 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:19:39.980613 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:19:39.980966 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:19:39.980971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:19:39.980974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:19:39.981629 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:19:39.981918 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 02:19:39.982380 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 02:19:39.982787 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8108,"thread_start_us":7876,"threads_started":1}
I20260928 02:19:39.990655 16009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4263109436924d6f914469ced887072b" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36211 } }
I20260928 02:19:39.991026 16009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:19:39.991110 16009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4263109436924d6f914469ced887072b, State: Initialized, Role: FOLLOWER
I20260928 02:19:39.991726 16009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4263109436924d6f914469ced887072b" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36211 } }
I20260928 02:19:39.991990 16009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:19:39.992111 16009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:19:39.992241 16009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:19:39.994405 16009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4263109436924d6f914469ced887072b" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36211 } }
I20260928 02:19:39.994864 16009 leader_election.cc:302] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4263109436924d6f914469ced887072b; no voters:
I20260928 02:19:39.995456 16009 leader_election.cc:288] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:19:39.995615 16014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:19:39.997062 16014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b [term 1 LEADER]: Becoming Leader. State: Replica: 4263109436924d6f914469ced887072b, State: Running, Role: LEADER
I20260928 02:19:39.997709 16014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4263109436924d6f914469ced887072b" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36211 } }
I20260928 02:19:39.998008 16009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:19:40.000504 16016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4263109436924d6f914469ced887072b. Latest consensus state: current_term: 1 leader_uuid: "4263109436924d6f914469ced887072b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4263109436924d6f914469ced887072b" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36211 } } }
I20260928 02:19:40.000871 16016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b [sys.catalog]: This master's current role is: LEADER
I20260928 02:19:40.001214 16015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4263109436924d6f914469ced887072b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4263109436924d6f914469ced887072b" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 36211 } } }
I20260928 02:19:40.001562 16015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b [sys.catalog]: This master's current role is: LEADER
I20260928 02:19:40.002450 16021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:19:40.010659 16021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:19:40.011484 16009 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:19:40.029623 16021 catalog_manager.cc:1347] Generated new cluster ID: edb8ede9f1894bd9819b00e985361b9d
I20260928 02:19:40.029783 16021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:19:40.046758 16021 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:19:40.048837 16021 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:19:40.056492 16021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b: Generated new TSK 0
I20260928 02:19:40.057801 16021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
I20260928 02:19:40.884963 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.140.212:39781/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:36211
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:19:41 dist-test-slave-h37r krb5kdc[15498](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790561981, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 02:19:41.290968 16054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:19:41.291320 16054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:19:41.291478 16054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:19:41.300143 16054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:19:41.300288 16054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:19:41.300450 16054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:19:41.311156 16054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.140.212:39781/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:36211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:19:41.312642 16054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:19:41.314244 16054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:19:41.332877 16064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:19:41.339327 16054 server_base.cc:1034] running on GCE node
W20260928 02:19:41.339360 16065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:19:41.340021 16067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:19:41.340968 16054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:19:41.345575 16054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:19:41.346949 16054 hybrid_clock.cc:648] HybridClock initialized: now 1790561981346867 us; error 75 us; skew 500 ppm
I20260928 02:19:41.347373 16054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:19:41 dist-test-slave-h37r krb5kdc[15498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561981, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:19:41.351955 16054 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:19:41.360090 16054 webserver.cc:460] Webserver started at http://127.4.140.193:43397/ using document root <none> and password file <none>
I20260928 02:19:41.361096 16054 fs_manager.cc:359] Metadata directory not provided
I20260928 02:19:41.361239 16054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:19:41.361640 16054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:19:41 dist-test-slave-h37r krb5kdc[15498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561981, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
I20260928 02:19:41.380757 16054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "f21080aaf270462b8cb9663ea3828276"
format_stamp: "Formatted at 2026-09-28 02:19:41 on dist-test-slave-h37r"
server_key: "43d73d81fb6f8d3a8a3f351f2e857ce7"
server_key_iv: "3a7c5dcbe65ec0ff3e8221eb516fb717"
server_key_version: "kuduclusterkey@0"
I20260928 02:19:41.382081 16054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "f21080aaf270462b8cb9663ea3828276"
format_stamp: "Formatted at 2026-09-28 02:19:41 on dist-test-slave-h37r"
server_key: "43d73d81fb6f8d3a8a3f351f2e857ce7"
server_key_iv: "3a7c5dcbe65ec0ff3e8221eb516fb717"
server_key_version: "kuduclusterkey@0"
I20260928 02:19:41.388721 16054 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 02:19:41.416488 16074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:19:41.422124 16054 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.000s sys 0.004s
I20260928 02:19:41.422340 16054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "f21080aaf270462b8cb9663ea3828276"
format_stamp: "Formatted at 2026-09-28 02:19:41 on dist-test-slave-h37r"
server_key: "43d73d81fb6f8d3a8a3f351f2e857ce7"
server_key_iv: "3a7c5dcbe65ec0ff3e8221eb516fb717"
server_key_version: "kuduclusterkey@0"
I20260928 02:19:41.422614 16054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:19:41.472121 16054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:19:41.477530 16054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:19:41.478259 16054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:19:41.480412 16054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:19:41.484686 16054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:19:41.484839 16054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 02:19:41.485054 16054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:19:41.485210 16054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:19:41.601011 16054 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:34563
I20260928 02:19:41.602027 16186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:34563 every 8 connection(s)
I20260928 02:19:41.602787 16054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:19:41.603994 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 16054
I20260928 02:19:41.604187 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:19:41.604986 4659 external_mini_cluster.cc:1428] Setting key 69fd17abd145a710a0151f3504af56cd
Sep 28 02:19:41 dist-test-slave-h37r krb5kdc[15498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561981, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.140.194@KRBTEST.COM; defaulting to no policy
I20260928 02:19:41.676538 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:41.608700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:35619 (local address 127.4.140.254:36211)
0928 02:19:41.645451 (+ 36751us) server_negotiation.cc:206] Beginning negotiation
0928 02:19:41.645458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:19:41.647994 (+ 2536us) server_negotiation.cc:407] Connection header received
0928 02:19:41.650174 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:19:41.650181 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:19:41.650271 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:19:41.650353 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:19:41.653318 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:41.654363 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:41.656705 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:41.658228 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:41.658347 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:41.668100 (+ 9753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:19:41.668110 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:19:41.668115 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:19:41.668162 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:19:41.671666 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:19:41.672717 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:19:41.672724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:19:41.672727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:19:41.672883 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:19:41.673879 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:19:41.673885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:19:41.673889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:19:41.674442 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:19:41.674589 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:19:41.675689 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0928 02:19:41.676148 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36598,"thread_start_us":36747,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.194@KRBTEST.COM" created.
Sep 28 02:19:41 dist-test-slave-h37r krb5kdc[15498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561981, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
I20260928 02:19:41.689332 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:41.607256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36211 (local address 127.4.140.193:35619)
0928 02:19:41.646200 (+ 38944us) negotiation.cc:107] Waiting for socket to connect
0928 02:19:41.646268 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 02:19:41.648904 (+ 2636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:19:41.650936 (+ 2032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:19:41.650949 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:19:41.651648 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:19:41.652500 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:19:41.652523 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:19:41.654582 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:19:41.654586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:19:41.656493 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:19:41.656515 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:19:41.660107 (+ 3592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:19:41.660116 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:19:41.660424 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:41.664141 (+ 3717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:19:41.664173 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:19:41.666978 (+ 2805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:19:41.671883 (+ 4905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:19:41.671921 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:19:41.671942 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:19:41.672559 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:19:41.673438 (+ 879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:19:41.673444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:19:41.673448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:19:41.673735 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:19:41.674692 (+ 957us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:19:41.674700 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:19:41.675186 (+ 486us) client_negotiation.cc:769] Sending connection context
0928 02:19:41.686303 (+ 11117us) client_negotiation.cc:240] Negotiation successful
0928 02:19:41.687007 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38568,"thread_start_us":41826,"threads_started":1}
I20260928 02:19:41.691581 16187 heartbeater.cc:347] Connected to a master server at 127.4.140.254:36211
I20260928 02:19:41.691957 16187 heartbeater.cc:464] Registering TS with master...
I20260928 02:19:41.692888 16187 heartbeater.cc:511] Master 127.4.140.254:36211 requested a full tablet report, sending...
I20260928 02:19:41.695429 15974 ts_manager.cc:194] Registered new tserver with Master: f21080aaf270462b8cb9663ea3828276 (127.4.140.193:34563)
I20260928 02:19:41.697656 15974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:35619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
W20260928 02:19:41.727322 4659 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.4.140.194: real 0.116s user 0.000s sys 0.019s
I20260928 02:19:41.728389 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.140.212:39781/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:36211
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:19:42 dist-test-slave-h37r krb5kdc[15498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561981, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
W20260928 02:19:42.266914 16193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:19:42.267438 16193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:19:42.267637 16193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:19:42.279004 16193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:19:42.279639 16193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:19:42.279922 16193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:19:42.295492 16193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.140.212:39781/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:36211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:19:42.297515 16193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:19:42.299284 16193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:19:42.329960 16201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:19:42.332321 16200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:19:42.356921 16193 server_base.cc:1034] running on GCE node
W20260928 02:19:42.358673 16203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:19:42.359638 16193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:19:42.366299 16193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:19:42.371145 16193 hybrid_clock.cc:648] HybridClock initialized: now 1790561982371057 us; error 71 us; skew 500 ppm
I20260928 02:19:42.371613 16193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:19:42 dist-test-slave-h37r krb5kdc[15498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561982, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:19:42.376129 16193 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:19:42.397527 16193 webserver.cc:460] Webserver started at http://127.4.140.194:34319/ using document root <none> and password file <none>
I20260928 02:19:42.398758 16193 fs_manager.cc:359] Metadata directory not provided
I20260928 02:19:42.398998 16193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:19:42.399456 16193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:19:42 dist-test-slave-h37r krb5kdc[15498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561982, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
I20260928 02:19:42.435030 16193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "fb3d98ffba264584bd39779abdec8a50"
format_stamp: "Formatted at 2026-09-28 02:19:42 on dist-test-slave-h37r"
server_key: "dbd66abc3747abff354ac9c943b7dd5b"
server_key_iv: "a86fa42750233b9b628e97c427037b6b"
server_key_version: "kuduclusterkey@0"
I20260928 02:19:42.436375 16193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "fb3d98ffba264584bd39779abdec8a50"
format_stamp: "Formatted at 2026-09-28 02:19:42 on dist-test-slave-h37r"
server_key: "dbd66abc3747abff354ac9c943b7dd5b"
server_key_iv: "a86fa42750233b9b628e97c427037b6b"
server_key_version: "kuduclusterkey@0"
I20260928 02:19:42.450062 16193 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.011s sys 0.000s
I20260928 02:19:42.501102 16210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:19:42.506207 16193 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.003s sys 0.002s
I20260928 02:19:42.506642 16193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "fb3d98ffba264584bd39779abdec8a50"
format_stamp: "Formatted at 2026-09-28 02:19:42 on dist-test-slave-h37r"
server_key: "dbd66abc3747abff354ac9c943b7dd5b"
server_key_iv: "a86fa42750233b9b628e97c427037b6b"
server_key_version: "kuduclusterkey@0"
I20260928 02:19:42.507001 16193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:19:42.566286 16193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:19:42.572642 16193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:19:42.573035 16193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:19:42.580441 16193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:19:42.589367 16193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:19:42.589556 16193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:19:42.589726 16193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:19:42.589828 16193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:19:42.701673 16187 heartbeater.cc:503] Master 127.4.140.254:36211 was elected leader, sending a full tablet report...
I20260928 02:19:42.870329 16193 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:44307
I20260928 02:19:42.870417 16329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:44307 every 8 connection(s)
I20260928 02:19:42.872141 16193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:19:42.872887 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 16193
I20260928 02:19:42.873059 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:19:42.873561 4659 external_mini_cluster.cc:1428] Setting key f1fc40961d6d81d51f60e3e3699df771
Sep 28 02:19:42 dist-test-slave-h37r krb5kdc[15498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561982, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.140.195@KRBTEST.COM; defaulting to no policy
I20260928 02:19:42.918802 16333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:42.884080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:59715 (local address 127.4.140.254:36211)
0928 02:19:42.895660 (+ 11580us) server_negotiation.cc:206] Beginning negotiation
0928 02:19:42.895699 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0928 02:19:42.897168 (+ 1469us) server_negotiation.cc:407] Connection header received
0928 02:19:42.899706 (+ 2538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:19:42.899713 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:19:42.899798 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:19:42.899870 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:19:42.902096 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:42.903234 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:42.905946 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:42.906592 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:42.906690 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:42.911089 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:19:42.911096 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:19:42.911101 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:19:42.911142 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:19:42.914756 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:19:42.915715 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:19:42.915721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:19:42.915725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:19:42.915866 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:19:42.916518 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:19:42.916522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:19:42.916526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:19:42.916865 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:19:42.917020 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:19:42.918146 (+ 1126us) server_negotiation.cc:299] Negotiation successful
0928 02:19:42.918369 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11280,"thread_start_us":11411,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.195@KRBTEST.COM" created.
I20260928 02:19:42.919952 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:42.877711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36211 (local address 127.4.140.194:59715)
0928 02:19:42.896718 (+ 19007us) negotiation.cc:107] Waiting for socket to connect
0928 02:19:42.896783 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 02:19:42.899336 (+ 2553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:19:42.900440 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:19:42.900454 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:19:42.901046 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:19:42.901886 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:19:42.901913 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:19:42.903439 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:19:42.903444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:19:42.905784 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:19:42.905799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:19:42.907046 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:19:42.907053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:19:42.907335 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:42.908134 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:19:42.908164 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:19:42.910749 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:19:42.914927 (+ 4178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:19:42.914937 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:19:42.914956 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:19:42.915574 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:19:42.916074 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:19:42.916079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:19:42.916083 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:19:42.916376 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:19:42.917010 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:19:42.917019 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:19:42.917553 (+ 534us) client_negotiation.cc:769] Sending connection context
0928 02:19:42.917927 (+ 374us) client_negotiation.cc:240] Negotiation successful
0928 02:19:42.918482 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18390,"thread_start_us":18129,"threads_started":1}
I20260928 02:19:42.925623 16330 heartbeater.cc:347] Connected to a master server at 127.4.140.254:36211
I20260928 02:19:42.925980 16330 heartbeater.cc:464] Registering TS with master...
I20260928 02:19:42.926940 16330 heartbeater.cc:511] Master 127.4.140.254:36211 requested a full tablet report, sending...
I20260928 02:19:42.928872 15974 ts_manager.cc:194] Registered new tserver with Master: fb3d98ffba264584bd39779abdec8a50 (127.4.140.194:44307)
I20260928 02:19:42.929942 15974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:59715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
I20260928 02:19:42.955670 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.140.212:39781/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:36211
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:19:43.297425 16336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:19:43.297766 16336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:19:43.297868 16336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:19:43.305228 16336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:19:43.305336 16336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:19:43.305466 16336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:19:43.315809 16336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.140.212:39781/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:36211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:19:43.317353 16336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:19:43.318572 16336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:19:43.332398 16341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:19:43.332432 16344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:19:43.332376 16342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:19:43.333277 16336 server_base.cc:1034] running on GCE node
I20260928 02:19:43.333916 16336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:19:43.334853 16336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:19:43.336058 16336 hybrid_clock.cc:648] HybridClock initialized: now 1790561983336014 us; error 33 us; skew 500 ppm
I20260928 02:19:43.336377 16336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:19:43 dist-test-slave-h37r krb5kdc[15498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:19:43.339437 16336 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:19:43.340811 16336 webserver.cc:460] Webserver started at http://127.4.140.195:45365/ using document root <none> and password file <none>
I20260928 02:19:43.341451 16336 fs_manager.cc:359] Metadata directory not provided
I20260928 02:19:43.341553 16336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:19:43.341866 16336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:19:43 dist-test-slave-h37r krb5kdc[15498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
I20260928 02:19:43.358960 16336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "5f86952e98ff4acb89dc81aa1e7847ba"
format_stamp: "Formatted at 2026-09-28 02:19:43 on dist-test-slave-h37r"
server_key: "4869cbd5407ed8c787c29f42e022e74d"
server_key_iv: "694452b721c4030351afa7ae75aca640"
server_key_version: "kuduclusterkey@0"
I20260928 02:19:43.360106 16336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "5f86952e98ff4acb89dc81aa1e7847ba"
format_stamp: "Formatted at 2026-09-28 02:19:43 on dist-test-slave-h37r"
server_key: "4869cbd5407ed8c787c29f42e022e74d"
server_key_iv: "694452b721c4030351afa7ae75aca640"
server_key_version: "kuduclusterkey@0"
I20260928 02:19:43.365818 16336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 02:19:43.382927 16351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:19:43.384580 16336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:19:43.384774 16336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "5f86952e98ff4acb89dc81aa1e7847ba"
format_stamp: "Formatted at 2026-09-28 02:19:43 on dist-test-slave-h37r"
server_key: "4869cbd5407ed8c787c29f42e022e74d"
server_key_iv: "694452b721c4030351afa7ae75aca640"
server_key_version: "kuduclusterkey@0"
I20260928 02:19:43.385008 16336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:19:43.403972 16336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:19:43.408170 16336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:19:43.408417 16336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:19:43.409289 16336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:19:43.410557 16336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:19:43.410630 16336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:19:43.410719 16336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:19:43.410810 16336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:19:43.444293 16336 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:38263
I20260928 02:19:43.444321 16463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:38263 every 8 connection(s)
I20260928 02:19:43.445640 16336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:19:43.453293 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 16336
I20260928 02:19:43.453462 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:19:43.453979 4659 external_mini_cluster.cc:1428] Setting key 6243e1ff6a54f2edade8b568ca08cd67
Sep 28 02:19:43 dist-test-slave-h37r krb5kdc[15498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:19:43.465139 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:43.448273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:60759 (local address 127.4.140.254:36211)
0928 02:19:43.448685 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0928 02:19:43.448690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:19:43.449730 (+ 1040us) server_negotiation.cc:407] Connection header received
0928 02:19:43.451475 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:19:43.451479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:19:43.451561 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:19:43.451668 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:19:43.453186 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.454247 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.456258 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.456843 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.456978 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:43.460154 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:19:43.460159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:19:43.460162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:19:43.460192 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:19:43.462525 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:19:43.463223 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:19:43.463226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:19:43.463229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:19:43.463345 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:19:43.463708 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:19:43.463711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:19:43.463713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:19:43.464019 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:19:43.464131 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:19:43.464732 (+ 601us) server_negotiation.cc:299] Negotiation successful
0928 02:19:43.464897 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":214,"threads_started":1}
I20260928 02:19:43.466104 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:43.448851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:36211 (local address 127.4.140.195:60759)
0928 02:19:43.449573 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0928 02:19:43.449613 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:19:43.451213 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:19:43.451873 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:19:43.451881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:19:43.452440 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:19:43.453019 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:19:43.453037 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.454418 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.454422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:19:43.456092 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:19:43.456111 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.457055 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.457063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:19:43.457305 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:43.457973 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:19:43.457995 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:19:43.459969 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:19:43.462667 (+ 2698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:19:43.462673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:19:43.462685 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:19:43.463114 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:19:43.463431 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:19:43.463434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:19:43.463436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:19:43.463617 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:19:43.464129 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:19:43.464135 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:19:43.464441 (+ 306us) client_negotiation.cc:769] Sending connection context
0928 02:19:43.464651 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 02:19:43.465024 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":304,"threads_started":1}
I20260928 02:19:43.467486 16464 heartbeater.cc:347] Connected to a master server at 127.4.140.254:36211
I20260928 02:19:43.467814 16464 heartbeater.cc:464] Registering TS with master...
I20260928 02:19:43.468577 16464 heartbeater.cc:511] Master 127.4.140.254:36211 requested a full tablet report, sending...
I20260928 02:19:43.469925 15974 ts_manager.cc:194] Registered new tserver with Master: 5f86952e98ff4acb89dc81aa1e7847ba (127.4.140.195:38263)
I20260928 02:19:43.470718 15974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:60759
I20260928 02:19:43.481299 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:19:43.498544 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:43.486156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46298 (local address 127.4.140.254:36211)
0928 02:19:43.486348 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:19:43.486357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:19:43.486688 (+ 331us) server_negotiation.cc:407] Connection header received
0928 02:19:43.486893 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:19:43.486903 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:19:43.486975 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:19:43.487072 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:19:43.488649 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.489404 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.490817 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.491215 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.491349 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:43.493700 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:19:43.493711 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:19:43.493715 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:19:43.493745 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:19:43.496013 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:19:43.496630 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:19:43.496636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:19:43.496639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:19:43.496769 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:19:43.497121 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:19:43.497128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:19:43.497131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:19:43.497402 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:19:43.497651 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 02:19:43.498015 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 02:19:43.498231 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:19:43.502353 15974 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:19:43.509555 15974 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:19:43.538097 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:43.523179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59278 (local address 127.4.140.193:34563)
0928 02:19:43.524200 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0928 02:19:43.524213 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:19:43.524922 (+ 709us) server_negotiation.cc:407] Connection header received
0928 02:19:43.525069 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:19:43.525081 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:19:43.525407 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:19:43.525633 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:19:43.526586 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.528778 (+ 2192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.532618 (+ 3840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.534727 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.534919 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:43.535245 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0928 02:19:43.537547 (+ 2302us) server_negotiation.cc:299] Negotiation successful
0928 02:19:43.537725 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":646,"threads_started":1}
I20260928 02:19:43.538106 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:43.522679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:34563 (local address 127.0.0.1:59278)
0928 02:19:43.524641 (+ 1962us) negotiation.cc:107] Waiting for socket to connect
0928 02:19:43.524660 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:19:43.524783 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:19:43.525933 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:19:43.525942 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:19:43.525977 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:19:43.526363 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:19:43.526374 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.529002 (+ 2628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.529011 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:19:43.532354 (+ 3343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:19:43.532379 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.537021 (+ 4642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.537026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:19:43.537250 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:43.537309 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 02:19:43.537442 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 02:19:43.537652 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1840,"thread_start_us":1724,"threads_started":1}
I20260928 02:19:43.542781 16122 tablet_service.cc:1467] Processing CreateTablet for tablet 587ad469c2c841f9af381bacdf7f109b (DEFAULT_TABLE table=test-table [id=24efaf52260f41f388d53d56c09a3999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:19:43.544483 16122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587ad469c2c841f9af381bacdf7f109b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:19:43.545409 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:43.525129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60496 (local address 127.4.140.195:38263)
0928 02:19:43.537515 (+ 12386us) server_negotiation.cc:206] Beginning negotiation
0928 02:19:43.537522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:19:43.537551 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:19:43.537640 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:19:43.537647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:19:43.537902 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:19:43.538797 (+ 895us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:19:43.539029 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.540368 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.542311 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.543771 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.544524 (+ 753us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:43.544763 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 02:19:43.544893 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 02:19:43.545061 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12215,"thread_start_us":12054,"threads_started":1}
I20260928 02:19:43.545411 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:43.524782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:38263 (local address 127.0.0.1:60496)
0928 02:19:43.536711 (+ 11929us) negotiation.cc:107] Waiting for socket to connect
0928 02:19:43.536737 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:19:43.536888 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:19:43.538131 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:19:43.538138 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:19:43.538166 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:19:43.538600 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:19:43.538610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.540527 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.540532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:19:43.542163 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:19:43.542182 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.544061 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.544068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:19:43.544666 (+ 598us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:43.544707 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 02:19:43.544833 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 02:19:43.544978 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11749,"mutex_wait_us":5,"thread_start_us":11571,"threads_started":1}
I20260928 02:19:43.545917 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:43.522588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.4.140.194:44307)
0928 02:19:43.536252 (+ 13664us) server_negotiation.cc:206] Beginning negotiation
0928 02:19:43.536265 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:19:43.536304 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:19:43.536424 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:19:43.536433 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:19:43.536722 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:19:43.538242 (+ 1520us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:19:43.538477 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.540025 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.542574 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.543962 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.544081 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:43.544671 (+ 590us) server_negotiation.cc:1035] Waiting for connection context
0928 02:19:43.545514 (+ 843us) server_negotiation.cc:299] Negotiation successful
0928 02:19:43.545693 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13439,"thread_start_us":13333,"threads_started":1}
I20260928 02:19:43.546293 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:43.522428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:44307 (local address 127.0.0.1:55586)
0928 02:19:43.523503 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0928 02:19:43.523562 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 02:19:43.523790 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:19:43.537615 (+ 13825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:19:43.537619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:19:43.537645 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:19:43.537992 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:19:43.538003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.540193 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.540197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:19:43.542411 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:19:43.542426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.544840 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.544846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:19:43.545009 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:43.545035 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:19:43.545125 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 02:19:43.545261 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":26,"thread_start_us":595,"threads_started":1}
I20260928 02:19:43.549233 16399 tablet_service.cc:1467] Processing CreateTablet for tablet 587ad469c2c841f9af381bacdf7f109b (DEFAULT_TABLE table=test-table [id=24efaf52260f41f388d53d56c09a3999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:19:43.549603 16265 tablet_service.cc:1467] Processing CreateTablet for tablet 587ad469c2c841f9af381bacdf7f109b (DEFAULT_TABLE table=test-table [id=24efaf52260f41f388d53d56c09a3999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:19:43.550801 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587ad469c2c841f9af381bacdf7f109b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:19:43.550806 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587ad469c2c841f9af381bacdf7f109b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:19:43.557982 16483 tablet_bootstrap.cc:492] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276: Bootstrap starting.
I20260928 02:19:43.561586 16483 tablet_bootstrap.cc:654] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276: Neither blocks nor log segments found. Creating new log.
I20260928 02:19:43.563076 16483 log.cc:824] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276: Log is configured to *not* fsync() on all Append() calls
I20260928 02:19:43.565258 16485 tablet_bootstrap.cc:492] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba: Bootstrap starting.
I20260928 02:19:43.566399 16483 tablet_bootstrap.cc:492] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276: No bootstrap required, opened a new log
I20260928 02:19:43.566043 16486 tablet_bootstrap.cc:492] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50: Bootstrap starting.
I20260928 02:19:43.566763 16483 ts_tablet_manager.cc:1397] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260928 02:19:43.569295 16485 tablet_bootstrap.cc:654] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba: Neither blocks nor log segments found. Creating new log.
I20260928 02:19:43.570129 16486 tablet_bootstrap.cc:654] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50: Neither blocks nor log segments found. Creating new log.
I20260928 02:19:43.570551 16485 log.cc:824] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba: Log is configured to *not* fsync() on all Append() calls
I20260928 02:19:43.571276 16486 log.cc:824] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50: Log is configured to *not* fsync() on all Append() calls
I20260928 02:19:43.573360 16485 tablet_bootstrap.cc:492] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba: No bootstrap required, opened a new log
I20260928 02:19:43.573609 16485 ts_tablet_manager.cc:1397] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260928 02:19:43.574139 16486 tablet_bootstrap.cc:492] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50: No bootstrap required, opened a new log
I20260928 02:19:43.574383 16486 ts_tablet_manager.cc:1397] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 02:19:43.577922 16483 raft_consensus.cc:348] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3d98ffba264584bd39779abdec8a50" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 44307 } } peers { permanent_uuid: "f21080aaf270462b8cb9663ea3828276" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 34563 } } peers { permanent_uuid: "5f86952e98ff4acb89dc81aa1e7847ba" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38263 } }
I20260928 02:19:43.578373 16483 raft_consensus.cc:374] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:19:43.578523 16483 raft_consensus.cc:729] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21080aaf270462b8cb9663ea3828276, State: Initialized, Role: FOLLOWER
I20260928 02:19:43.579146 16483 consensus_queue.cc:260] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3d98ffba264584bd39779abdec8a50" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 44307 } } peers { permanent_uuid: "f21080aaf270462b8cb9663ea3828276" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 34563 } } peers { permanent_uuid: "5f86952e98ff4acb89dc81aa1e7847ba" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38263 } }
I20260928 02:19:43.580904 16483 ts_tablet_manager.cc:1428] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260928 02:19:43.582190 16485 raft_consensus.cc:348] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3d98ffba264584bd39779abdec8a50" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 44307 } } peers { permanent_uuid: "f21080aaf270462b8cb9663ea3828276" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 34563 } } peers { permanent_uuid: "5f86952e98ff4acb89dc81aa1e7847ba" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38263 } }
I20260928 02:19:43.582983 16485 raft_consensus.cc:374] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:19:43.583086 16485 raft_consensus.cc:729] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f86952e98ff4acb89dc81aa1e7847ba, State: Initialized, Role: FOLLOWER
I20260928 02:19:43.583657 16485 consensus_queue.cc:260] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3d98ffba264584bd39779abdec8a50" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 44307 } } peers { permanent_uuid: "f21080aaf270462b8cb9663ea3828276" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 34563 } } peers { permanent_uuid: "5f86952e98ff4acb89dc81aa1e7847ba" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38263 } }
I20260928 02:19:43.584789 16464 heartbeater.cc:503] Master 127.4.140.254:36211 was elected leader, sending a full tablet report...
I20260928 02:19:43.584710 16486 raft_consensus.cc:348] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3d98ffba264584bd39779abdec8a50" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 44307 } } peers { permanent_uuid: "f21080aaf270462b8cb9663ea3828276" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 34563 } } peers { permanent_uuid: "5f86952e98ff4acb89dc81aa1e7847ba" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38263 } }
I20260928 02:19:43.585122 16486 raft_consensus.cc:374] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:19:43.585209 16486 raft_consensus.cc:729] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb3d98ffba264584bd39779abdec8a50, State: Initialized, Role: FOLLOWER
I20260928 02:19:43.585434 16485 ts_tablet_manager.cc:1428] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260928 02:19:43.585855 16486 consensus_queue.cc:260] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3d98ffba264584bd39779abdec8a50" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 44307 } } peers { permanent_uuid: "f21080aaf270462b8cb9663ea3828276" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 34563 } } peers { permanent_uuid: "5f86952e98ff4acb89dc81aa1e7847ba" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38263 } }
I20260928 02:19:43.587883 16330 heartbeater.cc:503] Master 127.4.140.254:36211 was elected leader, sending a full tablet report...
I20260928 02:19:43.588220 16486 ts_tablet_manager.cc:1428] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260928 02:19:43.627242 16331 tablet.cc:2369] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:19:43.655072 16188 tablet.cc:2369] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:19:43.664635 16489 raft_consensus.cc:482] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:19:43.664929 16489 raft_consensus.cc:504] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3d98ffba264584bd39779abdec8a50" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 44307 } } peers { permanent_uuid: "f21080aaf270462b8cb9663ea3828276" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 34563 } } peers { permanent_uuid: "5f86952e98ff4acb89dc81aa1e7847ba" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38263 } }
I20260928 02:19:43.666563 16489 leader_election.cc:288] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb3d98ffba264584bd39779abdec8a50 (127.4.140.194:44307), 5f86952e98ff4acb89dc81aa1e7847ba (127.4.140.195:38263)
I20260928 02:19:43.673624 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:43.667087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:44367 (local address 127.4.140.195:38263)
0928 02:19:43.667300 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 02:19:43.667306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:19:43.668373 (+ 1067us) server_negotiation.cc:407] Connection header received
0928 02:19:43.668531 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:19:43.668537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:19:43.668612 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:19:43.668706 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:19:43.669312 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.670067 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.671867 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.672667 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.672809 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:43.672926 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 02:19:43.673225 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 02:19:43.673369 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:19:43.673678 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:43.667106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:47011 (local address 127.4.140.194:44307)
0928 02:19:43.667290 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:19:43.667297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:19:43.668551 (+ 1254us) server_negotiation.cc:407] Connection header received
0928 02:19:43.668718 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:19:43.668724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:19:43.668814 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:19:43.668913 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:19:43.669511 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.670236 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.671989 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.672709 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.672946 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:43.673023 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:19:43.673318 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 02:19:43.673451 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:19:43.673727 16493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:43.667452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:38263 (local address 127.4.140.193:44367)
0928 02:19:43.668265 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0928 02:19:43.668281 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:19:43.668399 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:19:43.668730 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:19:43.668736 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:19:43.668786 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:19:43.669159 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:19:43.669170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.670204 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.670211 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:19:43.671706 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:19:43.671717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.672898 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.672904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:19:43.673080 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:43.673103 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:19:43.673176 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:19:43.673293 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":55,"thread_start_us":518,"threads_started":1}
I20260928 02:19:43.673805 16492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:43.667449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:44307 (local address 127.4.140.193:47011)
0928 02:19:43.668414 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0928 02:19:43.668439 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:19:43.668581 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:19:43.668952 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:19:43.668955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:19:43.668974 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:19:43.669355 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:19:43.669368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.670385 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.670388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:19:43.671811 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:19:43.671827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.673017 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.673021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:19:43.673137 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:43.673185 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 02:19:43.673279 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 02:19:43.673390 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"mutex_wait_us":42,"thread_start_us":447,"threads_started":1}
I20260928 02:19:43.674757 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587ad469c2c841f9af381bacdf7f109b" candidate_uuid: "f21080aaf270462b8cb9663ea3828276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f86952e98ff4acb89dc81aa1e7847ba" is_pre_election: true
I20260928 02:19:43.674769 16285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587ad469c2c841f9af381bacdf7f109b" candidate_uuid: "f21080aaf270462b8cb9663ea3828276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb3d98ffba264584bd39779abdec8a50" is_pre_election: true
I20260928 02:19:43.675326 16285 raft_consensus.cc:2454] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21080aaf270462b8cb9663ea3828276 in term 0.
I20260928 02:19:43.675326 16419 raft_consensus.cc:2454] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21080aaf270462b8cb9663ea3828276 in term 0.
I20260928 02:19:43.676025 16078 leader_election.cc:302] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f21080aaf270462b8cb9663ea3828276, fb3d98ffba264584bd39779abdec8a50; no voters:
I20260928 02:19:43.676417 16489 raft_consensus.cc:2781] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:19:43.676975 16489 raft_consensus.cc:482] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:19:43.677091 16489 raft_consensus.cc:3037] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:19:43.679163 16489 raft_consensus.cc:504] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3d98ffba264584bd39779abdec8a50" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 44307 } } peers { permanent_uuid: "f21080aaf270462b8cb9663ea3828276" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 34563 } } peers { permanent_uuid: "5f86952e98ff4acb89dc81aa1e7847ba" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38263 } }
I20260928 02:19:43.679761 16489 leader_election.cc:288] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [CANDIDATE]: Term 1 election: Requested vote from peers fb3d98ffba264584bd39779abdec8a50 (127.4.140.194:44307), 5f86952e98ff4acb89dc81aa1e7847ba (127.4.140.195:38263)
I20260928 02:19:43.680163 16285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587ad469c2c841f9af381bacdf7f109b" candidate_uuid: "f21080aaf270462b8cb9663ea3828276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb3d98ffba264584bd39779abdec8a50"
I20260928 02:19:43.680184 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587ad469c2c841f9af381bacdf7f109b" candidate_uuid: "f21080aaf270462b8cb9663ea3828276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f86952e98ff4acb89dc81aa1e7847ba"
I20260928 02:19:43.680357 16285 raft_consensus.cc:3037] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:19:43.680373 16419 raft_consensus.cc:3037] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:19:43.682452 16285 raft_consensus.cc:2454] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21080aaf270462b8cb9663ea3828276 in term 1.
I20260928 02:19:43.682502 16419 raft_consensus.cc:2454] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21080aaf270462b8cb9663ea3828276 in term 1.
I20260928 02:19:43.682811 16078 leader_election.cc:302] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f21080aaf270462b8cb9663ea3828276, fb3d98ffba264584bd39779abdec8a50; no voters:
I20260928 02:19:43.683072 16489 raft_consensus.cc:2781] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:19:43.683539 16489 raft_consensus.cc:686] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [term 1 LEADER]: Becoming Leader. State: Replica: f21080aaf270462b8cb9663ea3828276, State: Running, Role: LEADER
I20260928 02:19:43.684297 16489 consensus_queue.cc:237] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3d98ffba264584bd39779abdec8a50" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 44307 } } peers { permanent_uuid: "f21080aaf270462b8cb9663ea3828276" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 34563 } } peers { permanent_uuid: "5f86952e98ff4acb89dc81aa1e7847ba" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38263 } }
I20260928 02:19:43.688951 15973 catalog_manager.cc:5495] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 reported cstate change: term changed from 0 to 1, leader changed from <none> to f21080aaf270462b8cb9663ea3828276 (127.4.140.193). New cstate: current_term: 1 leader_uuid: "f21080aaf270462b8cb9663ea3828276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3d98ffba264584bd39779abdec8a50" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 44307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f21080aaf270462b8cb9663ea3828276" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f86952e98ff4acb89dc81aa1e7847ba" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38263 } health_report { overall_health: UNKNOWN } } }
W20260928 02:19:43.697124 16465 tablet.cc:2369] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:19:43.739071 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:43.733521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59286 (local address 127.4.140.193:34563)
0928 02:19:43.734222 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0928 02:19:43.734228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:19:43.734255 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:19:43.734346 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:19:43.734350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:19:43.734435 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:19:43.734559 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:19:43.734899 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.735603 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.737070 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.737478 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.737570 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:43.738026 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:19:43.738529 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:19:43.738632 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:19:43.738707 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 02:19:43.738782 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:19:43.747239 16285 raft_consensus.cc:1261] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50 [term 1 FOLLOWER]: Refusing update from remote peer f21080aaf270462b8cb9663ea3828276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:19:43.747239 16419 raft_consensus.cc:1261] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba [term 1 FOLLOWER]: Refusing update from remote peer f21080aaf270462b8cb9663ea3828276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:19:43.748343 16494 consensus_queue.cc:1035] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f86952e98ff4acb89dc81aa1e7847ba" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 38263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:19:43.748705 16489 consensus_queue.cc:1035] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb3d98ffba264584bd39779abdec8a50" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 44307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:19:43.763875 16498 mvcc.cc:205] Tried to move back new op lower bound from 7334141885417881600 to 7334141885174280192. Current Snapshot: MvccSnapshot[applied={T|T < 7334141885417881600}]
I20260928 02:19:43.765679 16499 mvcc.cc:205] Tried to move back new op lower bound from 7334141885417881600 to 7334141885174280192. Current Snapshot: MvccSnapshot[applied={T|T < 7334141885417881600 or (T in {7334141885417881600})}]
I20260928 02:19:43.774169 15973 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:19:43.774439 15973 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:19:43.777758 15973 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4263109436924d6f914469ced887072b: Sending DeleteTablet for 3 replicas of tablet 587ad469c2c841f9af381bacdf7f109b
I20260928 02:19:43.778774 16265 tablet_service.cc:1514] Processing DeleteTablet for tablet 587ad469c2c841f9af381bacdf7f109b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:19:43 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:55586
I20260928 02:19:43.779078 16122 tablet_service.cc:1514] Processing DeleteTablet for tablet 587ad469c2c841f9af381bacdf7f109b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:19:43 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:59278
I20260928 02:19:43.779204 16399 tablet_service.cc:1514] Processing DeleteTablet for tablet 587ad469c2c841f9af381bacdf7f109b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:19:43 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:60496
I20260928 02:19:43.780745 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 16054
I20260928 02:19:43.793457 16508 tablet_replica.cc:330] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba: stopping tablet replica
I20260928 02:19:43.793759 16506 tablet_replica.cc:330] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50: stopping tablet replica
I20260928 02:19:43.794018 16508 raft_consensus.cc:2229] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:19:43.794234 16506 raft_consensus.cc:2229] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:19:43.794692 16508 raft_consensus.cc:2258] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:19:43.794799 16506 raft_consensus.cc:2258] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:19:43.797132 16508 ts_tablet_manager.cc:1905] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:19:43.797135 16506 ts_tablet_manager.cc:1905] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:19:43.798589 16507 tablet_replica.cc:330] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276: stopping tablet replica
I20260928 02:19:43.799099 16507 raft_consensus.cc:2229] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [term 1 LEADER]: Raft consensus shutting down.
Sep 28 02:19:43 dist-test-slave-h37r krb5kdc[15498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:19:43.804395 16506 ts_tablet_manager.cc:1918] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:19:43.804551 16508 ts_tablet_manager.cc:1918] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:19:43.804584 16506 log.cc:1196] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/587ad469c2c841f9af381bacdf7f109b
I20260928 02:19:43.804800 16508 log.cc:1196] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/587ad469c2c841f9af381bacdf7f109b
I20260928 02:19:43.805060 16507 raft_consensus.cc:2258] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:19:43.805150 16506 ts_tablet_manager.cc:1939] T 587ad469c2c841f9af381bacdf7f109b P fb3d98ffba264584bd39779abdec8a50: Deleting consensus metadata
I20260928 02:19:43.805410 16508 ts_tablet_manager.cc:1939] T 587ad469c2c841f9af381bacdf7f109b P 5f86952e98ff4acb89dc81aa1e7847ba: Deleting consensus metadata
I20260928 02:19:43.807205 15960 catalog_manager.cc:4841] TS 5f86952e98ff4acb89dc81aa1e7847ba (127.4.140.195:38263): tablet 587ad469c2c841f9af381bacdf7f109b (table test-table [id=24efaf52260f41f388d53d56c09a3999]) successfully deleted
I20260928 02:19:43.807531 16507 ts_tablet_manager.cc:1905] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:19:43.807714 15961 catalog_manager.cc:4841] TS fb3d98ffba264584bd39779abdec8a50 (127.4.140.194:44307): tablet 587ad469c2c841f9af381bacdf7f109b (table test-table [id=24efaf52260f41f388d53d56c09a3999]) successfully deleted
I20260928 02:19:43.814002 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:43.783473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59288 (local address 127.4.140.193:34563)
0928 02:19:43.783701 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 02:19:43.783715 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:19:43.793217 (+ 9502us) server_negotiation.cc:407] Connection header received
0928 02:19:43.793480 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:19:43.793486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:19:43.793580 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:19:43.793691 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:19:43.795782 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.797055 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.799105 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.799837 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.800001 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:43.805946 (+ 5945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:19:43.805962 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:19:43.805985 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:19:43.806050 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:19:43.810368 (+ 4318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:19:43.811380 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:19:43.811389 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:19:43.811393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:19:43.811688 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:19:43.812187 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:19:43.812192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:19:43.812196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:19:43.812699 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:19:43.813036 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 02:19:43.813391 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 02:19:43.813639 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:19:43.814347 16507 ts_tablet_manager.cc:1918] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:19:43.814513 16507 log.cc:1196] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/587ad469c2c841f9af381bacdf7f109b
I20260928 02:19:43.815536 16507 ts_tablet_manager.cc:1939] T 587ad469c2c841f9af381bacdf7f109b P f21080aaf270462b8cb9663ea3828276: Deleting consensus metadata
I20260928 02:19:43.815881 16182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:19:43.926100 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 16054
I20260928 02:19:43.926123 15960 catalog_manager.cc:4841] TS f21080aaf270462b8cb9663ea3828276 (127.4.140.193:34563): tablet 587ad469c2c841f9af381bacdf7f109b (table test-table [id=24efaf52260f41f388d53d56c09a3999]) successfully deleted
I20260928 02:19:43.944615 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 16193
Sep 28 02:19:43 dist-test-slave-h37r krb5kdc[15498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.194@KRBTEST.COM
I20260928 02:19:43.960696 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:43.945415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55594 (local address 127.4.140.194:44307)
0928 02:19:43.945616 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 02:19:43.945624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:19:43.945661 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:19:43.945858 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:19:43.945865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:19:43.945938 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:19:43.946039 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:19:43.947285 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.948160 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.949519 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:43.949943 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:43.950083 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:43.954607 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:19:43.954629 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:19:43.954646 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:19:43.954682 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:19:43.957822 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:19:43.958546 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:19:43.958561 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:19:43.958564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:19:43.958697 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:19:43.959076 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:19:43.959083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:19:43.959087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:19:43.959464 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:19:43.959758 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 02:19:43.960103 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 02:19:43.960334 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:19:43.961315 16325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:19:44.051069 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 16193
I20260928 02:19:44.067366 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 16336
Sep 28 02:19:44 dist-test-slave-h37r krb5kdc[15498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.195@KRBTEST.COM
I20260928 02:19:44.081954 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:19:44.068150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.4.140.195:38263)
0928 02:19:44.068343 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:19:44.068347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:19:44.068366 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:19:44.068523 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:19:44.068527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:19:44.068590 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:19:44.068674 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:19:44.069890 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:44.070611 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:44.071884 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:19:44.072346 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:19:44.072440 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:19:44.076742 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:19:44.076757 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:19:44.076774 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:19:44.076811 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:19:44.079380 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:19:44.080151 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:19:44.080159 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:19:44.080161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:19:44.080304 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:19:44.080650 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:19:44.080654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:19:44.080655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:19:44.081035 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:19:44.081237 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 02:19:44.081522 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 02:19:44.081686 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:19:44.082597 16459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:19:44.171584 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 16336
I20260928 02:19:44.188234 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 15942
I20260928 02:19:44.189492 16004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:19:44.299979 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 15942
I20260928 02:19:44.322190 4659 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 02:19:44.657799 4659 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 02:19:44.658008 4659 mini_ranger.cc:66] Stopping Ranger...
I20260928 02:19:45.003831 4659 mini_ranger.cc:68] Stopped Ranger
2026-09-28 02:19:45.004 UTC [15528] LOG: received smart shutdown request
2026-09-28 02:19:45.007 UTC [15528] LOG: background worker "logical replication launcher" (PID 15537) exited with exit code 1
2026-09-28 02:19:45.008 UTC [15532] LOG: shutting down
2026-09-28 02:19:45.180 UTC [15528] LOG: database system is shut down
2026-09-28T02:19:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49311 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:19:45 dist-test-slave-h37r krb5kdc[16522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:19:45 dist-test-slave-h37r krb5kdc[16522](info): set up 2 sockets
Sep 28 02:19:45 dist-test-slave-h37r krb5kdc[16522](info): commencing operation
krb5kdc: starting...
W20260928 02:19:46.420241 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:19:46 dist-test-slave-h37r krb5kdc[16522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:19:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:19:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.140.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.140.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.140.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/rangeradmin_127.4.140.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.140.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/rangeradmin_127.4.140.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.140.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.140.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.140.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/rangerlookup_127.4.140.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.140.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/rangerlookup_127.4.140.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.140.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.140.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.140.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/HTTP_127.4.140.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.140.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/HTTP_127.4.140.212@KRBTEST.COM.keytab.
I20260928 02:19:46.738976 4659 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskx4rqyh/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/postgres -l logfile start
2026-09-28 02:19:49.290 UTC [16552] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:19:49.290 UTC [16552] LOG: listening on IPv4 address "127.4.140.212", port 36105
2026-09-28 02:19:49.294 UTC [16552] LOG: listening on Unix socket "/tmp/.s.PGSQL.36105"
2026-09-28 02:19:49.316 UTC [16555] LOG: database system was shut down at 2026-09-28 02:19:47 UTC
2026-09-28 02:19:49.319 UTC [16552] LOG: database system is ready to accept connections
I20260928 02:19:50.291976 4659 mini_postgres.cc:96] Postgres bound to 36105
2026-09-28 02:19:50.309 UTC [16563] FATAL: database "slave" does not exist
127.4.140.212:36105 - accepting connections
I20260928 02:19:50.309942 4659 mini_ranger.cc:161] Starting Ranger...
I20260928 02:19:50.335887 4659 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:19:50.824971 4659 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:19:50.825084 4659 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ranger-admin
2026-09-28 02:19:51,118 [I] DB FLAVOR :POSTGRES
2026-09-28 02:19:51,119 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:19:51,119 [I] Checking connection..
2026-09-28 02:19:51,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:19:51,356 [I] Checking connection passed.
2026-09-28 02:19:51,356 [I] --------- Verifying version history table ---------
2026-09-28 02:19:51,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:19:51,643 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:19:51,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:19:51,906 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:19:51,906 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:19:51,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:19:52.133 UTC [16627] WARNING: there is no transaction in progress
2026-09-28 02:19:52,140 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:19:52,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:19:52,400 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:19:52,400 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:19:52,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:19:52,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:19:52,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-h37r.c.gcp-upstream.internal','N') ;"
2026-09-28 02:19:53,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:19:53,415 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:19:53,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:19:53,662 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:19:53,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:19:53,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:19:54,146 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:19:54,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:19:55.133 UTC [16735] WARNING: there is no transaction in progress
2026-09-28 02:19:55.149 UTC [16735] WARNING: there is no transaction in progress
2026-09-28 02:19:55.166 UTC [16735] WARNING: there is no transaction in progress
2026-09-28 02:19:55.181 UTC [16735] WARNING: there is no transaction in progress
2026-09-28 02:19:55.197 UTC [16735] WARNING: there is no transaction in progress
2026-09-28 02:19:55.215 UTC [16735] WARNING: there is no transaction in progress
2026-09-28 02:19:55.280 UTC [16735] WARNING: there is no transaction in progress
2026-09-28 02:19:55.287 UTC [16735] WARNING: there is no transaction in progress
2026-09-28 02:19:55.301 UTC [16735] WARNING: there is no transaction in progress
2026-09-28 02:19:55.311 UTC [16735] WARNING: there is no transaction in progress
2026-09-28 02:19:55.322 UTC [16735] WARNING: there is no transaction in progress
2026-09-28 02:19:55.335 UTC [16735] WARNING: there is no transaction in progress
2026-09-28 02:19:55.345 UTC [16735] WARNING: there is no transaction in progress
2026-09-28 02:19:55.352 UTC [16735] WARNING: there is no transaction in progress
2026-09-28 02:19:55.359 UTC [16735] WARNING: there is no transaction in progress
2026-09-28 02:19:55.365 UTC [16735] WARNING: there is no transaction in progress
2026-09-28 02:19:56,015 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:19:56,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:19:56,263 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:19:56,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:19:56,495 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:19:56,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:19:56,735 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:19:56,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:19:56,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:19:57,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-h37r.c.gcp-upstream.internal';"
2026-09-28 02:19:57,506 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:19:57,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:19:57,749 [I] DB_PATCHES have already been applied
I20260928 02:19:57.755765 4659 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ranger-admin:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskx4rqyh/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:19:57.755877 4659 mini_ranger.cc:193] Using host: 127.4.140.212
I20260928 02:19:57.770895 4659 mini_ranger.cc:227] Ranger admin URL: http://127.4.140.212:46079
Sep 28, 2026 2:19:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:19:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:19:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:19:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:19:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:19:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:19:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:19:59.003039 4659 mini_ranger.cc:160] Time spent starting Ranger: real 8.693s user 0.008s sys 0.032s
Sep 28, 2026 2:20:19 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:20:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:20:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:20:19 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:20:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:20:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:20:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:20:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:20:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:20:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:20:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:20:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:20:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:20:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:20:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:20:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:20:19 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:20:20 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 02:20:21.380137 4659 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.140.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.140.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.140.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/rangerkms_127.4.140.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.140.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/rangerkms_127.4.140.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.140.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.140.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.140.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/HTTP_127.4.140.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.140.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/HTTP_127.4.140.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 02:20:21 dist-test-slave-h37r krb5kdc[16522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562021, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 02:20:21.559077 4659 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 02:20:21.580338 4659 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 02:20:22.526333 4659 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 02:20:22.526453 4659 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ranger-kms
I20260928 02:20:22.526492 4659 mini_ranger_kms.cc:227] Using postgres at 127.4.140.212:36105
2026-09-28 02:20:22,883 [I] DB FLAVOR :POSTGRES
2026-09-28 02:20:22,884 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:20:22,884 [I] Checking connection
2026-09-28 02:20:22,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 02:20:23,131 [I] connection success
2026-09-28 02:20:23,131 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 02:20:23,131 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 02:20:23,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 02:20:23,360 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 02:20:23,361 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:20:23,361 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 02:20:23,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.140.212:36105/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 02:20:23,616 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 02:20:24.085690 4659 mini_ranger_kms.cc:310] Created kms service
I20260928 02:20:24.277557 4659 mini_ranger_kms.cc:326] Created kudu user
I20260928 02:20:24.321707 4659 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 02:20:24 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562021, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
I20260928 02:20:24.720551 4659 mini_ranger_kms.cc:384] Added ranger policy
I20260928 02:20:24.720867 4659 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ranger-kms:/tmp/dist-test-taskx4rqyh/build/asan/bin/postgresql.jar:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskx4rqyh/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 02:20:24.720937 4659 mini_ranger_kms.cc:240] Using host: 127.4.140.212
I20260928 02:20:24.742002 4659 mini_ranger_kms.cc:276] Ranger KMS PID: 16916
I20260928 02:20:24.742132 4659 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.140.212:39753
Sep 28, 2026 2:20:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:20:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:20:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:20:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:20:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:20:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 02:20:27 dist-test-slave-h37r krb5kdc[16522](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790562027, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.140.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:27.128692 4659 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.570s user 0.102s sys 0.270s
I20260928 02:20:27.128968 4659 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 02:20:27.129189 4659 mini_ranger_kms.cc:401] 127.4.140.212:39753/kms/v1/keys
Sep 28 02:20:27 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562027, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.140.212@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
Sep 28 02:20:27 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562027, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.140.212@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
Sep 28 02:20:30 dist-test-slave-h37r krb5kdc[16522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:20:30.334293 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:45447
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:42501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:45447
--ranger_config_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.140.212:39753/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:20:30.661885 16966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:30.662228 16966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:30.662386 16966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:30.669883 16966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:20:30.670054 16966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:30.670158 16966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:30.670274 16966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:20:30.670358 16966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:20:30.680666 16966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:42501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.140.212:39753/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:45447
--ranger_config_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:45447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:30.682024 16966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:30.683430 16966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:30.697824 16972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:30.698136 16971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:30.698372 16974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:30.698827 16966 server_base.cc:1034] running on GCE node
I20260928 02:20:30.699577 16966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:30.701128 16966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:30.702348 16966 hybrid_clock.cc:648] HybridClock initialized: now 1790562030702306 us; error 36 us; skew 500 ppm
I20260928 02:20:30.702729 16966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:30 dist-test-slave-h37r krb5kdc[16522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562030, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:30.714833 16966 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:20:30.716635 16966 webserver.cc:460] Webserver started at http://127.4.140.254:35543/ using document root <none> and password file <none>
I20260928 02:20:30.717325 16966 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:30.717420 16966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:20:30.717762 16966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:20:30 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562030, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
I20260928 02:20:30.777019 16966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "16e21b80676f4b398843b3d3755f7021"
format_stamp: "Formatted at 2026-09-28 02:20:30 on dist-test-slave-h37r"
server_key: "7e09584efcee038d8df48612e88774f7"
server_key_iv: "24ea96b9e19345ad5c1ddcec9dbe29c9"
server_key_version: "kuduclusterkey@0"
I20260928 02:20:30.777937 16966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "16e21b80676f4b398843b3d3755f7021"
format_stamp: "Formatted at 2026-09-28 02:20:30 on dist-test-slave-h37r"
server_key: "7e09584efcee038d8df48612e88774f7"
server_key_iv: "24ea96b9e19345ad5c1ddcec9dbe29c9"
server_key_version: "kuduclusterkey@0"
I20260928 02:20:30.783646 16966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:20:30.803580 16981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:30.805476 16966 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 02:20:30.805684 16966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "16e21b80676f4b398843b3d3755f7021"
format_stamp: "Formatted at 2026-09-28 02:20:30 on dist-test-slave-h37r"
server_key: "7e09584efcee038d8df48612e88774f7"
server_key_iv: "24ea96b9e19345ad5c1ddcec9dbe29c9"
server_key_version: "kuduclusterkey@0"
I20260928 02:20:30.805985 16966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:30.836823 16966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:30.841922 16966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:30.842314 16966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:30.871167 16966 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:45447
I20260928 02:20:30.871224 17032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:45447 every 8 connection(s)
I20260928 02:20:30.872728 16966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:20:30.877915 17033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:20:30.882121 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 16966
I20260928 02:20:30.882371 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:20:30.883025 4659 external_mini_cluster.cc:1428] Setting key 54237264d6c429a7a7deac38c2ad5edd
I20260928 02:20:30.893784 17033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021: Bootstrap starting.
I20260928 02:20:30.898468 17033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021: Neither blocks nor log segments found. Creating new log.
I20260928 02:20:30.900098 17033 log.cc:824] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021: Log is configured to *not* fsync() on all Append() calls
I20260928 02:20:30.904067 17033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021: No bootstrap required, opened a new log
Sep 28 02:20:30 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:20:30.917613 17033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e21b80676f4b398843b3d3755f7021" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45447 } }
I20260928 02:20:30.918139 17033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:20:30.918260 17033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16e21b80676f4b398843b3d3755f7021, State: Initialized, Role: FOLLOWER
I20260928 02:20:30.919005 17033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e21b80676f4b398843b3d3755f7021" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45447 } }
I20260928 02:20:30.919294 17033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:20:30.919435 17033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:20:30.919677 17033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:20:30.921641 17036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:30.885499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52374 (local address 127.4.140.254:45447)
0928 02:20:30.900603 (+ 15104us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:30.900615 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:30.900664 (+ 49us) server_negotiation.cc:407] Connection header received
0928 02:20:30.902805 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:30.902828 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:30.903201 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:30.903591 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:30.905010 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:30.906342 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:30.907674 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:30.908443 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:30.908656 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:30.912956 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:30.912964 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:30.912979 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:30.913014 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:30.917329 (+ 4315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:30.918017 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:30.918026 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:30.918036 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:30.918239 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:30.918717 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:30.918722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:30.918725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:30.919298 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:30.919576 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:30.920102 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 02:20:30.920643 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14869,"thread_start_us":14587,"threads_started":1}
I20260928 02:20:30.923084 17033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e21b80676f4b398843b3d3755f7021" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45447 } }
I20260928 02:20:30.923709 17033 leader_election.cc:302] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16e21b80676f4b398843b3d3755f7021; no voters:
I20260928 02:20:30.924867 17033 leader_election.cc:288] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:20:30.924933 17038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:20:30.927117 17033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:20:30.928201 17038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [term 1 LEADER]: Becoming Leader. State: Replica: 16e21b80676f4b398843b3d3755f7021, State: Running, Role: LEADER
I20260928 02:20:30.928802 17038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e21b80676f4b398843b3d3755f7021" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45447 } }
I20260928 02:20:30.932735 17039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16e21b80676f4b398843b3d3755f7021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e21b80676f4b398843b3d3755f7021" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45447 } } }
I20260928 02:20:30.933073 17039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [sys.catalog]: This master's current role is: LEADER
I20260928 02:20:30.933606 17040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16e21b80676f4b398843b3d3755f7021. Latest consensus state: current_term: 1 leader_uuid: "16e21b80676f4b398843b3d3755f7021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e21b80676f4b398843b3d3755f7021" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45447 } } }
I20260928 02:20:30.934060 17040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [sys.catalog]: This master's current role is: LEADER
I20260928 02:20:30.944075 17043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:20:30.952483 17043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:20:30.976214 17043 catalog_manager.cc:1347] Generated new cluster ID: 7a777ca5c0a14bc9a93f343ffbd12bf8
I20260928 02:20:30.976369 17043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:20:31.001922 17043 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:20:31.004621 17043 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:20:31.006209 17033 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:20:31.029065 17043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021: Generated new TSK 0
I20260928 02:20:31.030485 17043 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
I20260928 02:20:31.789737 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.140.212:39753/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:45447
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:42501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:20:32 dist-test-slave-h37r krb5kdc[16522](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790562032, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 02:20:32.278131 17079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:32.278539 17079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:32.278755 17079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:32.290069 17079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:32.290292 17079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:32.290503 17079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:20:32.303495 17079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:42501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.140.212:39753/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:45447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:32.304951 17079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:32.306245 17079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:32.324150 17090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:32.325234 17092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:32.326354 17089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:32.326929 17079 server_base.cc:1034] running on GCE node
I20260928 02:20:32.327833 17079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:32.329203 17079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:32.330503 17079 hybrid_clock.cc:648] HybridClock initialized: now 1790562032330437 us; error 47 us; skew 500 ppm
I20260928 02:20:32.330910 17079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:32 dist-test-slave-h37r krb5kdc[16522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562032, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:32.335165 17079 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:20:32.340080 17079 webserver.cc:460] Webserver started at http://127.4.140.193:39897/ using document root <none> and password file <none>
I20260928 02:20:32.341025 17079 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:32.341173 17079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:20:32.341530 17079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:20:32 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562032, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
I20260928 02:20:32.360769 17079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "da0ff51095e243de80f05b608a903447"
format_stamp: "Formatted at 2026-09-28 02:20:32 on dist-test-slave-h37r"
server_key: "9bf577eec9ed27f5da7e18b64f07e903"
server_key_iv: "97bc50b0360cd808d951e79130ad6265"
server_key_version: "kuduclusterkey@0"
I20260928 02:20:32.362697 17079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "da0ff51095e243de80f05b608a903447"
format_stamp: "Formatted at 2026-09-28 02:20:32 on dist-test-slave-h37r"
server_key: "9bf577eec9ed27f5da7e18b64f07e903"
server_key_iv: "97bc50b0360cd808d951e79130ad6265"
server_key_version: "kuduclusterkey@0"
I20260928 02:20:32.390218 17079 fs_manager.cc:691] Time spent creating directory manager: real 0.027s user 0.014s sys 0.011s
I20260928 02:20:32.427798 17099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:32.430140 17079 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.008s sys 0.014s
I20260928 02:20:32.430439 17079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "da0ff51095e243de80f05b608a903447"
format_stamp: "Formatted at 2026-09-28 02:20:32 on dist-test-slave-h37r"
server_key: "9bf577eec9ed27f5da7e18b64f07e903"
server_key_iv: "97bc50b0360cd808d951e79130ad6265"
server_key_version: "kuduclusterkey@0"
I20260928 02:20:32.430824 17079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:32.473824 17079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:32.478960 17079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:32.479350 17079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:32.484560 17079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:20:32.486392 17079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:20:32.486488 17079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:32.486635 17079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:20:32.486703 17079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:20:32 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562032, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
I20260928 02:20:32.600317 17079 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:44869
I20260928 02:20:32.600375 17211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:44869 every 8 connection(s)
I20260928 02:20:32.602124 17079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:20:32.608299 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 17079
I20260928 02:20:32.608547 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:20:32.609447 4659 external_mini_cluster.cc:1428] Setting key b1df5dc4e3c70ddff054329c652dc329
Sep 28 02:20:32 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562032, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:20:32.658384 17214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:32.612768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:45447 (local address 127.4.140.193:59581)
0928 02:20:32.624155 (+ 11387us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:32.624225 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:32.626838 (+ 2613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:32.634370 (+ 7532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:32.634383 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:32.635061 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:32.635971 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:32.636001 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:32.637799 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:32.637804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:32.639875 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:32.639890 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:32.641163 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:32.641169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:32.641447 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:32.642273 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:32.642308 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:32.648599 (+ 6291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:32.653089 (+ 4490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:32.653100 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:32.653122 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:32.653759 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:32.654274 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:32.654280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:32.654285 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:32.654561 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:32.655314 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:32.655326 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:32.655852 (+ 526us) client_negotiation.cc:769] Sending connection context
0928 02:20:32.656234 (+ 382us) client_negotiation.cc:240] Negotiation successful
0928 02:20:32.656829 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10992,"thread_start_us":14415,"threads_started":1}
WARNING: no policy specified for kudu/127.4.140.194@KRBTEST.COM; defaulting to no policy
I20260928 02:20:32.660259 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:32.615540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:59581 (local address 127.4.140.254:45447)
0928 02:20:32.633618 (+ 18078us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:32.633625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:32.633648 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:20:32.633718 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:32.633726 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:32.633813 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:32.633929 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:32.636236 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:32.637601 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:32.640093 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:32.640734 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:32.640877 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:32.648876 (+ 7999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:32.648885 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:32.648894 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:32.648944 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:32.652907 (+ 3963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:32.653938 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:32.653946 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:32.653953 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:32.654125 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:32.654730 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:32.654735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:32.654738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:32.655131 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:32.659405 (+ 4274us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:32.659564 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 02:20:32.659815 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17928,"thread_start_us":17780,"threads_started":1}
I20260928 02:20:32.661950 17212 heartbeater.cc:347] Connected to a master server at 127.4.140.254:45447
I20260928 02:20:32.662282 17212 heartbeater.cc:464] Registering TS with master...
I20260928 02:20:32.663101 17212 heartbeater.cc:511] Master 127.4.140.254:45447 requested a full tablet report, sending...
I20260928 02:20:32.665591 16998 ts_manager.cc:194] Registered new tserver with Master: da0ff51095e243de80f05b608a903447 (127.4.140.193:44869)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.194@KRBTEST.COM" created.
I20260928 02:20:32.667575 16998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:59581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
I20260928 02:20:32.700838 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.140.212:39753/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:45447
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:42501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:20:32 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562032, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
W20260928 02:20:33.223917 17218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:33.224427 17218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:33.224637 17218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:33.234259 17218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:33.234407 17218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:33.234596 17218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:20:33.245029 17218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:42501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.140.212:39753/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:45447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:33.246409 17218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:33.248181 17218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:33.264864 17232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:33.268564 17233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:33.269791 17235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:33.271780 17218 server_base.cc:1034] running on GCE node
I20260928 02:20:33.272545 17218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:33.273718 17218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:33.274962 17218 hybrid_clock.cc:648] HybridClock initialized: now 1790562033274885 us; error 62 us; skew 500 ppm
I20260928 02:20:33.275323 17218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:33 dist-test-slave-h37r krb5kdc[16522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562033, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:33.278658 17218 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:20:33.280668 17218 webserver.cc:460] Webserver started at http://127.4.140.194:34147/ using document root <none> and password file <none>
I20260928 02:20:33.281430 17218 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:33.281585 17218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:20:33.281927 17218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:20:33 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562033, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
I20260928 02:20:33.300792 17218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "f3755de48ad248f9a814abe9e1eaa563"
format_stamp: "Formatted at 2026-09-28 02:20:33 on dist-test-slave-h37r"
server_key: "3a0f0ce2d9f1cfa7ba081eab52b6a696"
server_key_iv: "ef6d9213571366f1a173727c6166177c"
server_key_version: "kuduclusterkey@0"
I20260928 02:20:33.302038 17218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "f3755de48ad248f9a814abe9e1eaa563"
format_stamp: "Formatted at 2026-09-28 02:20:33 on dist-test-slave-h37r"
server_key: "3a0f0ce2d9f1cfa7ba081eab52b6a696"
server_key_iv: "ef6d9213571366f1a173727c6166177c"
server_key_version: "kuduclusterkey@0"
I20260928 02:20:33.324153 17218 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.008s sys 0.011s
I20260928 02:20:33.356540 17242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:33.358685 17218 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.009s sys 0.004s
I20260928 02:20:33.358966 17218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "f3755de48ad248f9a814abe9e1eaa563"
format_stamp: "Formatted at 2026-09-28 02:20:33 on dist-test-slave-h37r"
server_key: "3a0f0ce2d9f1cfa7ba081eab52b6a696"
server_key_iv: "ef6d9213571366f1a173727c6166177c"
server_key_version: "kuduclusterkey@0"
I20260928 02:20:33.359302 17218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:33.420336 17218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:33.425494 17218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:33.425876 17218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:33.433316 17218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:20:33.441406 17218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:20:33.441574 17218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 02:20:33.441761 17218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:20:33.441860 17218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:33.535187 17218 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:41911
I20260928 02:20:33.535228 17354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:41911 every 8 connection(s)
I20260928 02:20:33.536911 17218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:20:33.539347 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 17218
I20260928 02:20:33.539523 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:20:33.540169 4659 external_mini_cluster.cc:1428] Setting key 102526c8f3dbe58d90223481789c8cbc
WARNING: no policy specified for kudu/127.4.140.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.195@KRBTEST.COM" created.
Sep 28 02:20:33 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562033, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:20:33.604161 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:33.543072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:55943 (local address 127.4.140.254:45447)
0928 02:20:33.565425 (+ 22353us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:33.565434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:33.569812 (+ 4378us) server_negotiation.cc:407] Connection header received
0928 02:20:33.569903 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:33.569910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:33.570020 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:33.570143 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:33.573516 (+ 3373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:33.574271 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:33.580036 (+ 5765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:33.580786 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:33.580939 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:33.596487 (+ 15548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:33.596498 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:33.596506 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:33.596565 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:33.600485 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:33.601451 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:33.601457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:33.601461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:33.601607 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:33.602124 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:33.602128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:33.602133 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:33.602555 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:33.602715 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:33.603483 (+ 768us) server_negotiation.cc:299] Negotiation successful
0928 02:20:33.603715 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22180,"thread_start_us":22382,"threads_started":1}
I20260928 02:20:33.605235 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:33.542504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:45447 (local address 127.4.140.194:55943)
0928 02:20:33.566195 (+ 23691us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:33.566264 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:33.569259 (+ 2995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:33.571536 (+ 2277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:33.571552 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:33.572350 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:33.573155 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:33.573176 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:33.574499 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:33.574503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:33.576561 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:33.576579 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:33.581265 (+ 4686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:33.581275 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:33.581616 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:33.582532 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:33.582568 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:33.596111 (+ 13543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:33.600669 (+ 4558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:33.600676 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:33.600691 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:33.601297 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:33.601775 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:33.601782 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:33.601787 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:33.601988 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:33.602722 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:33.602733 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:33.603095 (+ 362us) client_negotiation.cc:769] Sending connection context
0928 02:20:33.603370 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 02:20:33.603977 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23292,"thread_start_us":27670,"threads_started":1}
I20260928 02:20:33.607021 17355 heartbeater.cc:347] Connected to a master server at 127.4.140.254:45447
I20260928 02:20:33.607360 17355 heartbeater.cc:464] Registering TS with master...
I20260928 02:20:33.608222 17355 heartbeater.cc:511] Master 127.4.140.254:45447 requested a full tablet report, sending...
I20260928 02:20:33.609874 16998 ts_manager.cc:194] Registered new tserver with Master: f3755de48ad248f9a814abe9e1eaa563 (127.4.140.194:41911)
I20260928 02:20:33.610872 16998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:55943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
I20260928 02:20:33.614312 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.140.212:39753/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:45447
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:42501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:20:33.671185 17212 heartbeater.cc:503] Master 127.4.140.254:45447 was elected leader, sending a full tablet report...
W20260928 02:20:33.964982 17361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:33.965368 17361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:33.965554 17361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:33.973284 17361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:33.973461 17361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:33.973662 17361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:20:33.983820 17361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:42501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.140.212:39753/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:45447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:33.985234 17361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:33.986629 17361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:34.000242 17369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:34.000313 17367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:34.000293 17366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:34.005658 17361 server_base.cc:1034] running on GCE node
I20260928 02:20:34.006345 17361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:34.007342 17361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:34.008572 17361 hybrid_clock.cc:648] HybridClock initialized: now 1790562034008537 us; error 51 us; skew 500 ppm
I20260928 02:20:34.008922 17361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:34 dist-test-slave-h37r krb5kdc[16522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562034, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:34.012485 17361 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:20:34.013967 17361 webserver.cc:460] Webserver started at http://127.4.140.195:41545/ using document root <none> and password file <none>
I20260928 02:20:34.014693 17361 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:34.014837 17361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:20:34.015154 17361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:20:34 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562034, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
I20260928 02:20:34.029564 17361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "391d2c8b44ee4535bcef7dc62643e6c0"
format_stamp: "Formatted at 2026-09-28 02:20:34 on dist-test-slave-h37r"
server_key: "38803f6fb53c77f337ac8f839a107393"
server_key_iv: "85697614c2d056fd29edbc0d319d7023"
server_key_version: "kuduclusterkey@0"
I20260928 02:20:34.030411 17361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "391d2c8b44ee4535bcef7dc62643e6c0"
format_stamp: "Formatted at 2026-09-28 02:20:34 on dist-test-slave-h37r"
server_key: "38803f6fb53c77f337ac8f839a107393"
server_key_iv: "85697614c2d056fd29edbc0d319d7023"
server_key_version: "kuduclusterkey@0"
I20260928 02:20:34.035739 17361 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:20:34.053802 17376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:34.055500 17361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:20:34.055693 17361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "391d2c8b44ee4535bcef7dc62643e6c0"
format_stamp: "Formatted at 2026-09-28 02:20:34 on dist-test-slave-h37r"
server_key: "38803f6fb53c77f337ac8f839a107393"
server_key_iv: "85697614c2d056fd29edbc0d319d7023"
server_key_version: "kuduclusterkey@0"
I20260928 02:20:34.055982 17361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:34.074759 17361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:34.078737 17361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:34.079059 17361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:34.080063 17361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:20:34.081434 17361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:20:34.081506 17361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:34.081642 17361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:20:34.081691 17361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:34.115211 17361 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:46131
I20260928 02:20:34.115231 17488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:46131 every 8 connection(s)
I20260928 02:20:34.116581 17361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:20:34.118393 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 17361
I20260928 02:20:34.118566 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:20:34.119146 4659 external_mini_cluster.cc:1428] Setting key 12aa15459f165dd91d86a5a9b03a59b9
Sep 28 02:20:34 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562034, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:20:34.136942 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:34.119603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:57007 (local address 127.4.140.254:45447)
0928 02:20:34.120100 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:34.120104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:34.120956 (+ 852us) server_negotiation.cc:407] Connection header received
0928 02:20:34.123239 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:34.123245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:34.123349 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:34.123465 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:34.124922 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:34.125644 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:34.127204 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:34.127642 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:34.127812 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:34.130989 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:34.130993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:34.130997 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:34.131027 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:34.133860 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:34.134552 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:34.134557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:34.134561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:34.134696 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:34.135137 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:34.135143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:34.135146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:34.135430 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:34.135585 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:34.136339 (+ 754us) server_negotiation.cc:299] Negotiation successful
0928 02:20:34.136555 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":243,"threads_started":1}
I20260928 02:20:34.137885 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:34.119876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:45447 (local address 127.4.140.195:57007)
0928 02:20:34.120792 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:34.120845 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:34.122984 (+ 2139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:34.123708 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:34.123716 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:34.124202 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:34.124766 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:34.124782 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:34.125794 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:34.125799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:34.127056 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:34.127068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:34.127937 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:34.127945 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:34.128178 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:34.128797 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:34.128817 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:34.130782 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:34.133992 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:34.134001 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:34.134017 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:34.134427 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:34.134810 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:34.134815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:34.134819 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:34.135015 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:34.135570 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:34.135577 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:34.135941 (+ 364us) client_negotiation.cc:769] Sending connection context
0928 02:20:34.136227 (+ 286us) client_negotiation.cc:240] Negotiation successful
0928 02:20:34.136727 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":350,"threads_started":1}
I20260928 02:20:34.139220 17489 heartbeater.cc:347] Connected to a master server at 127.4.140.254:45447
I20260928 02:20:34.139602 17489 heartbeater.cc:464] Registering TS with master...
I20260928 02:20:34.140633 17489 heartbeater.cc:511] Master 127.4.140.254:45447 requested a full tablet report, sending...
I20260928 02:20:34.142045 16998 ts_manager.cc:194] Registered new tserver with Master: 391d2c8b44ee4535bcef7dc62643e6c0 (127.4.140.195:46131)
I20260928 02:20:34.142856 16998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:57007
I20260928 02:20:34.146961 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:20:34.148623 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 17079
Sep 28 02:20:34 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:20:34.164861 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:34.149555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56968 (local address 127.4.140.193:44869)
0928 02:20:34.150169 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:34.150178 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:34.150206 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:20:34.150273 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:34.150279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:34.150468 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:34.150636 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:34.152121 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:34.153101 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:34.154388 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:34.154899 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:34.154994 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:34.159294 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:34.159304 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:34.159316 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:34.159373 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:34.162443 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:34.163058 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:34.163064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:34.163067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:34.163205 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:34.163511 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:34.163514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:34.163516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:34.163827 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:34.164161 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:34.164382 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 02:20:34.164573 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":289,"threads_started":1}
I20260928 02:20:34.166152 17207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:20:34.249269 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 17079
I20260928 02:20:34.265151 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 17218
Sep 28 02:20:34 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.194@KRBTEST.COM
I20260928 02:20:34.280778 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:34.266055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36230 (local address 127.4.140.194:41911)
0928 02:20:34.266629 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:34.266637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:34.266662 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:20:34.266722 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:34.266728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:34.266942 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:34.267068 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:34.268252 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:34.269266 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:34.270700 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:34.271220 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:34.271318 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:34.275586 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:34.275596 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:34.275609 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:34.275645 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:34.278261 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:34.278849 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:34.278856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:34.278858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:34.278989 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:34.279293 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:34.279296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:34.279298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:34.279606 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:34.279804 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:34.280307 (+ 503us) server_negotiation.cc:299] Negotiation successful
0928 02:20:34.280499 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":326,"threads_started":1}
I20260928 02:20:34.282056 17350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:20:34.363977 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 17218
I20260928 02:20:34.380828 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 17361
Sep 28 02:20:34 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.195@KRBTEST.COM
I20260928 02:20:34.397492 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:34.381888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41544 (local address 127.4.140.195:46131)
0928 02:20:34.382538 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:34.382545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:34.382574 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:20:34.382632 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:34.382638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:34.382855 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:34.382981 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:34.384404 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:34.385470 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:34.386692 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:34.387234 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:34.387363 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:34.392056 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:34.392068 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:34.392083 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:34.392115 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:34.394927 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:34.395581 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:34.395587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:34.395589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:34.395715 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:34.396084 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:34.396088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:34.396090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:34.396436 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:34.396673 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:34.396887 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 02:20:34.397074 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":351,"threads_started":1}
I20260928 02:20:34.398759 17484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:20:34.481631 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 17361
I20260928 02:20:34.497462 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 16966
I20260928 02:20:34.498710 17028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:20:34.604458 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 16966
I20260928 02:20:34.628959 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:45447
--webserver_interface=127.4.140.254
--webserver_port=35543
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:42501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:45447
--ranger_config_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.140.212:39753/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.140.212:39753/kms with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:20:34.954660 17501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:34.954996 17501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:34.955171 17501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:34.962566 17501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:20:34.962721 17501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:34.962834 17501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:34.962930 17501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:20:34.963021 17501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:20:34.973367 17501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:42501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.140.212:39753/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:45447
--ranger_config_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:45447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=35543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:34.974644 17501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:34.976070 17501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:34.988319 17506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:34.988291 17507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:34.988238 17509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:34.989181 17501 server_base.cc:1034] running on GCE node
I20260928 02:20:34.989954 17501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:34.991421 17501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:34.992662 17501 hybrid_clock.cc:648] HybridClock initialized: now 1790562034992611 us; error 67 us; skew 500 ppm
I20260928 02:20:34.993021 17501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:34 dist-test-slave-h37r krb5kdc[16522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562034, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:34.996089 17501 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:20:34.997539 17501 webserver.cc:460] Webserver started at http://127.4.140.254:35543/ using document root <none> and password file <none>
I20260928 02:20:34.998221 17501 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:34.998370 17501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:20:35 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562034, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
I20260928 02:20:35.055248 17501 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:20:35.058706 17517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:35.060304 17501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:20:35.060508 17501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "16e21b80676f4b398843b3d3755f7021"
format_stamp: "Formatted at 2026-09-28 02:20:30 on dist-test-slave-h37r"
server_key: "7e09584efcee038d8df48612e88774f7"
server_key_iv: "24ea96b9e19345ad5c1ddcec9dbe29c9"
server_key_version: "kuduclusterkey@0"
I20260928 02:20:35.061501 17501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:35.084568 17501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:35.088419 17501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:35.088728 17501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:35.107677 17501 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:45447
I20260928 02:20:35.107685 17568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:45447 every 8 connection(s)
I20260928 02:20:35.109297 17501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:20:35.112375 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 17501
I20260928 02:20:35.113215 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.140.212:39753/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:44869
--local_ip_for_outbound_sockets=127.4.140.193
--tserver_master_addrs=127.4.140.254:45447
--webserver_port=39897
--webserver_interface=127.4.140.193
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:42501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.140.212:39753/kms with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:20:35.116612 17569 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:20:35.119872 17569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021: Bootstrap starting.
I20260928 02:20:35.138399 17569 log.cc:824] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021: Log is configured to *not* fsync() on all Append() calls
I20260928 02:20:35.147142 17569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:20:35.147663 17569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021: Bootstrap complete.
I20260928 02:20:35.156565 17569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e21b80676f4b398843b3d3755f7021" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45447 } }
I20260928 02:20:35.157431 17569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16e21b80676f4b398843b3d3755f7021, State: Initialized, Role: FOLLOWER
I20260928 02:20:35.158079 17569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e21b80676f4b398843b3d3755f7021" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45447 } }
I20260928 02:20:35.158320 17569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:20:35.158456 17569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:20:35.158635 17569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:20:35.162309 17569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e21b80676f4b398843b3d3755f7021" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45447 } }
I20260928 02:20:35.162786 17569 leader_election.cc:302] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16e21b80676f4b398843b3d3755f7021; no voters:
I20260928 02:20:35.163443 17569 leader_election.cc:288] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:20:35.163575 17573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:20:35.164691 17573 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [term 2 LEADER]: Becoming Leader. State: Replica: 16e21b80676f4b398843b3d3755f7021, State: Running, Role: LEADER
I20260928 02:20:35.165295 17573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e21b80676f4b398843b3d3755f7021" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45447 } }
I20260928 02:20:35.165688 17569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:20:35.167884 17574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "16e21b80676f4b398843b3d3755f7021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e21b80676f4b398843b3d3755f7021" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45447 } } }
I20260928 02:20:35.168334 17574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [sys.catalog]: This master's current role is: LEADER
I20260928 02:20:35.169382 17579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:20:35.168709 17575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16e21b80676f4b398843b3d3755f7021. Latest consensus state: current_term: 2 leader_uuid: "16e21b80676f4b398843b3d3755f7021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e21b80676f4b398843b3d3755f7021" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45447 } } }
I20260928 02:20:35.171391 17575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021 [sys.catalog]: This master's current role is: LEADER
I20260928 02:20:35.186796 17579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:20:35.188823 17579 catalog_manager.cc:1259] Loaded cluster ID: 7a777ca5c0a14bc9a93f343ffbd12bf8
I20260928 02:20:35.189000 17579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:20:35.193888 17579 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:20:35.195807 17579 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021: Loaded TSK: 0
I20260928 02:20:35.197732 17579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:20:35.201354 17569 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260928 02:20:35.561111 17571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:35.561594 17571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:35.561812 17571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:35.574013 17571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:35.574159 17571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:35.574393 17571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:20:35.590036 17571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:42501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.140.212:39753/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:44869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=39897
--tserver_master_addrs=127.4.140.254:45447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:35.591670 17571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:35.593432 17571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:35.644448 17607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:35.650535 17606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:35.659826 17609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:35.660404 17571 server_base.cc:1034] running on GCE node
I20260928 02:20:35.661233 17571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:35.682078 17571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:35.690989 17571 hybrid_clock.cc:648] HybridClock initialized: now 1790562035690864 us; error 159 us; skew 500 ppm
I20260928 02:20:35.691454 17571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:35 dist-test-slave-h37r krb5kdc[16522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562035, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:35.696154 17571 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:20:35.718153 17571 webserver.cc:460] Webserver started at http://127.4.140.193:39897/ using document root <none> and password file <none>
I20260928 02:20:35.719053 17571 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:35.719237 17571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:20:35 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562035, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
I20260928 02:20:35.754375 17571 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.004s sys 0.004s
I20260928 02:20:35.764816 17618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:35.766660 17571 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.008s
I20260928 02:20:35.766860 17571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "da0ff51095e243de80f05b608a903447"
format_stamp: "Formatted at 2026-09-28 02:20:32 on dist-test-slave-h37r"
server_key: "9bf577eec9ed27f5da7e18b64f07e903"
server_key_iv: "97bc50b0360cd808d951e79130ad6265"
server_key_version: "kuduclusterkey@0"
I20260928 02:20:35.768311 17571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:35.823108 17571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:35.828420 17571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:35.828759 17571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:35.829898 17571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:20:35.831398 17571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:20:35.831517 17571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:35.831647 17571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:20:35.831749 17571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:35.880396 17571 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:44869
I20260928 02:20:35.880438 17730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:44869 every 8 connection(s)
I20260928 02:20:35.881984 17571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:20:35.892544 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 17571
I20260928 02:20:35.893381 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.140.212:39753/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:41911
--local_ip_for_outbound_sockets=127.4.140.194
--tserver_master_addrs=127.4.140.254:45447
--webserver_port=34147
--webserver_interface=127.4.140.194
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:42501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.140.212:39753/kms with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:20:35 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562035, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:20:35.924280 17733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:35.886242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:45447 (local address 127.4.140.193:35533)
0928 02:20:35.890220 (+ 3978us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:35.890272 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:35.893535 (+ 3263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:35.899578 (+ 6043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:35.899595 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:35.900509 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:35.901560 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:35.901592 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:35.905288 (+ 3696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:35.905298 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:35.908118 (+ 2820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:35.908142 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:35.910115 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:35.910124 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:35.910527 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:35.911516 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:35.911556 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:35.914935 (+ 3379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:35.919315 (+ 4380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:35.919324 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:35.919340 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:35.919971 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:35.920475 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:35.920480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:35.920484 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:35.920753 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:35.921670 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:35.921678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:35.922077 (+ 399us) client_negotiation.cc:769] Sending connection context
0928 02:20:35.922357 (+ 280us) client_negotiation.cc:240] Negotiation successful
0928 02:20:35.922917 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3668,"thread_start_us":3409,"threads_started":1}
I20260928 02:20:35.927325 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:35.896491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:35533 (local address 127.4.140.254:45447)
0928 02:20:35.897422 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:35.897449 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:35.897511 (+ 62us) server_negotiation.cc:407] Connection header received
0928 02:20:35.898113 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:35.898139 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:35.898783 (+ 644us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:35.901934 (+ 3151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:35.902459 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:35.905025 (+ 2566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:35.908383 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:35.909780 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:35.911177 (+ 1397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:35.915234 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:35.915245 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:35.915282 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:35.915327 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:35.919138 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:35.920157 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:35.920166 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:35.920174 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:35.920348 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:35.920900 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:35.920905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:35.920909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:35.921520 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:35.925469 (+ 3949us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:35.925715 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 02:20:35.926127 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":3025,"threads_started":1}
I20260928 02:20:35.930944 17731 heartbeater.cc:347] Connected to a master server at 127.4.140.254:45447
I20260928 02:20:35.931259 17731 heartbeater.cc:464] Registering TS with master...
I20260928 02:20:35.932263 17731 heartbeater.cc:511] Master 127.4.140.254:45447 requested a full tablet report, sending...
W20260928 02:20:35.935585 17731 heartbeater.cc:644] Failed to heartbeat to 127.4.140.254:45447 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:20:35.935835 17731 heartbeater.cc:464] Registering TS with master...
I20260928 02:20:35.936188 17731 heartbeater.cc:511] Master 127.4.140.254:45447 requested a full tablet report, sending...
I20260928 02:20:35.937455 17731 heartbeater.cc:464] Registering TS with master...
I20260928 02:20:35.937760 17731 heartbeater.cc:511] Master 127.4.140.254:45447 requested a full tablet report, sending...
W20260928 02:20:35.938900 17731 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:20:36.341596 17735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:36.342005 17735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:36.342244 17735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:36.353849 17735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:36.353986 17735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:36.354212 17735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
Sep 28 02:20:36 dist-test-slave-h37r krb5kdc[16522](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790562036, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:36.369974 17735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:42501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.140.212:39753/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:41911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=34147
--tserver_master_addrs=127.4.140.254:45447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:36.371649 17735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:36.373548 17735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:36.400249 17751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:36.412951 17752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:36.413445 17735 server_base.cc:1034] running on GCE node
W20260928 02:20:36.413774 17755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:36.414575 17735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:36.415995 17735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:36.417303 17735 hybrid_clock.cc:648] HybridClock initialized: now 1790562036417227 us; error 56 us; skew 500 ppm
I20260928 02:20:36.417732 17735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:36 dist-test-slave-h37r krb5kdc[16522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562036, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:36.421770 17735 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:20:36.428306 17735 webserver.cc:460] Webserver started at http://127.4.140.194:34147/ using document root <none> and password file <none>
I20260928 02:20:36.429278 17735 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:36.429503 17735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:20:36 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562036, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
I20260928 02:20:36.459802 17735 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.003s sys 0.007s
I20260928 02:20:36.463845 17763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:36.465706 17735 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:20:36.465901 17735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "f3755de48ad248f9a814abe9e1eaa563"
format_stamp: "Formatted at 2026-09-28 02:20:33 on dist-test-slave-h37r"
server_key: "3a0f0ce2d9f1cfa7ba081eab52b6a696"
server_key_iv: "ef6d9213571366f1a173727c6166177c"
server_key_version: "kuduclusterkey@0"
I20260928 02:20:36.467319 17735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:36.509083 17735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:36.514503 17735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:36.514843 17735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:36.516230 17735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:20:36.518061 17735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:20:36.518180 17735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:36.518299 17735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:20:36.518368 17735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:36.577258 17735 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:41911
I20260928 02:20:36.577337 17875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:41911 every 8 connection(s)
I20260928 02:20:36.579244 17735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:20:36.580494 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 17735
I20260928 02:20:36.581331 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.140.212:39753/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:46131
--local_ip_for_outbound_sockets=127.4.140.195
--tserver_master_addrs=127.4.140.254:45447
--webserver_port=41545
--webserver_interface=127.4.140.195
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:42501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.140.212:39753/kms with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:20:36 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562036, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:20:36.623620 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:36.591914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:36395 (local address 127.4.140.254:45447)
0928 02:20:36.604328 (+ 12414us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:36.604335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:36.604360 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:20:36.604436 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:36.604441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:36.604540 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:36.604687 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:36.607053 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:36.608313 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:36.610900 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:36.611537 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:36.611637 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:36.616118 (+ 4481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:36.616124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:36.616128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:36.616167 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:36.619799 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:36.620795 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:36.620801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:36.620805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:36.620953 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:36.621515 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:36.621520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:36.621524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:36.621893 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:36.622013 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:36.623002 (+ 989us) server_negotiation.cc:299] Negotiation successful
0928 02:20:36.623226 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12260,"thread_start_us":12120,"threads_started":1}
I20260928 02:20:36.626201 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:36.592814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:45447 (local address 127.4.140.194:36395)
0928 02:20:36.596498 (+ 3684us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:36.596559 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:36.599023 (+ 2464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:36.605112 (+ 6089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:36.605127 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:36.605863 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:36.606790 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:36.606821 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:36.608549 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:36.608555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:36.610694 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:36.610709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:36.612055 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:36.612062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:36.612365 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:36.613199 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:36.613231 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:36.615818 (+ 2587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:36.620016 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:36.620025 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:36.620044 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:36.620633 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:36.621098 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:36.621103 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:36.621107 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:36.621379 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:36.622118 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:36.622126 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:36.622563 (+ 437us) client_negotiation.cc:769] Sending connection context
0928 02:20:36.624214 (+ 1651us) client_negotiation.cc:240] Negotiation successful
0928 02:20:36.624798 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3353,"thread_start_us":3099,"threads_started":1}
I20260928 02:20:36.628155 17876 heartbeater.cc:347] Connected to a master server at 127.4.140.254:45447
I20260928 02:20:36.628476 17876 heartbeater.cc:464] Registering TS with master...
I20260928 02:20:36.629345 17876 heartbeater.cc:511] Master 127.4.140.254:45447 requested a full tablet report, sending...
I20260928 02:20:36.631466 17534 ts_manager.cc:194] Registered new tserver with Master: f3755de48ad248f9a814abe9e1eaa563 (127.4.140.194:41911)
I20260928 02:20:36.633668 17534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:36395
Sep 28 02:20:36 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562036, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
I20260928 02:20:36.939080 17731 heartbeater.cc:464] Registering TS with master...
I20260928 02:20:36.939531 17731 heartbeater.cc:511] Master 127.4.140.254:45447 requested a full tablet report, sending...
I20260928 02:20:36.940917 17534 ts_manager.cc:194] Registered new tserver with Master: da0ff51095e243de80f05b608a903447 (127.4.140.193:44869)
I20260928 02:20:36.942143 17534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:35533
W20260928 02:20:37.005458 17878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:37.005790 17878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:37.005963 17878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:37.013797 17878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:37.013939 17878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:37.014127 17878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:20:37.024434 17878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:42501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.140.212:39753/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:46131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=41545
--tserver_master_addrs=127.4.140.254:45447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:37.025794 17878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:37.027071 17878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:37.043679 17885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:37.044332 17886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:37.048478 17888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:37.053753 17878 server_base.cc:1034] running on GCE node
I20260928 02:20:37.054545 17878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:37.056924 17878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:37.058171 17878 hybrid_clock.cc:648] HybridClock initialized: now 1790562037058106 us; error 53 us; skew 500 ppm
I20260928 02:20:37.058518 17878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:37 dist-test-slave-h37r krb5kdc[16522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562037, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:37.062700 17878 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:20:37.064678 17878 webserver.cc:460] Webserver started at http://127.4.140.195:41545/ using document root <none> and password file <none>
I20260928 02:20:37.065627 17878 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:37.065793 17878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:20:37 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562037, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
I20260928 02:20:37.096678 17878 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.007s sys 0.004s
I20260928 02:20:37.100788 17896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:37.102854 17878 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:20:37.103133 17878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "391d2c8b44ee4535bcef7dc62643e6c0"
format_stamp: "Formatted at 2026-09-28 02:20:34 on dist-test-slave-h37r"
server_key: "38803f6fb53c77f337ac8f839a107393"
server_key_iv: "85697614c2d056fd29edbc0d319d7023"
server_key_version: "kuduclusterkey@0"
I20260928 02:20:37.104667 17878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:37.147066 17878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:37.152901 17878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:37.153251 17878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:37.161535 17878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:20:37.163484 17878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:20:37.163599 17878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:37.163704 17878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:20:37.163771 17878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:20:37 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562036, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for HTTP/127.4.140.212@KRBTEST.COM
I20260928 02:20:37.543715 17878 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:46131
I20260928 02:20:37.543882 18016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:46131 every 8 connection(s)
I20260928 02:20:37.545321 17878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:20:37.548024 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 17878
Sep 28 02:20:37 dist-test-slave-h37r krb5kdc[16522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562037, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:20:37.573534 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:37.549093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37728 (local address 127.4.140.254:45447)
0928 02:20:37.549701 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:37.549711 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:37.555566 (+ 5855us) server_negotiation.cc:407] Connection header received
0928 02:20:37.555763 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:37.555769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:37.555868 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:37.556011 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:37.558313 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.559364 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.561410 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.562027 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.562166 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:37.565434 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:37.565441 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:37.565446 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:37.565494 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:37.570003 (+ 4509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:37.570776 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:37.570782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:37.570785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:37.570926 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:37.571541 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:37.571547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:37.571551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:37.572004 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:37.572393 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:37.572747 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 02:20:37.573085 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":263,"threads_started":1}
I20260928 02:20:37.574213 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:37.548323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:36635 (local address 127.4.140.254:45447)
0928 02:20:37.548867 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:37.548874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:37.552768 (+ 3894us) server_negotiation.cc:407] Connection header received
0928 02:20:37.554887 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:37.554893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:37.555010 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:37.555149 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:37.557708 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.558998 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.560972 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.561526 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.561652 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:37.566148 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:37.566157 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:37.566160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:37.566219 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:37.570098 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:37.571018 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:37.571026 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:37.571029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:37.571161 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:37.571684 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:37.571689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:37.571692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:37.572004 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:37.572173 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:37.573443 (+ 1270us) server_negotiation.cc:299] Negotiation successful
0928 02:20:37.573658 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":409,"spinlock_wait_cycles":1280,"thread_start_us":270,"threads_started":1}
I20260928 02:20:37.576769 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:37.548989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:45447 (local address 127.4.140.195:36635)
0928 02:20:37.552593 (+ 3604us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:37.552644 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:37.554588 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:37.555663 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:37.555680 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:37.556524 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:37.557492 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:37.557523 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.559192 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.559198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:37.560814 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:37.560827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.561783 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.561790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:37.562096 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:37.562954 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:37.562985 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:37.565867 (+ 2882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:37.570234 (+ 4367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:37.570244 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:37.570264 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:37.570877 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:37.571270 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:37.571276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:37.571280 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:37.571551 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:37.572193 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:37.572202 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:37.572667 (+ 465us) client_negotiation.cc:769] Sending connection context
0928 02:20:37.574666 (+ 1999us) client_negotiation.cc:240] Negotiation successful
0928 02:20:37.575294 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3286,"thread_start_us":3051,"threads_started":1}
I20260928 02:20:37.578598 18017 heartbeater.cc:347] Connected to a master server at 127.4.140.254:45447
I20260928 02:20:37.578918 18017 heartbeater.cc:464] Registering TS with master...
I20260928 02:20:37.579815 18017 heartbeater.cc:511] Master 127.4.140.254:45447 requested a full tablet report, sending...
I20260928 02:20:37.581490 17534 ts_manager.cc:194] Registered new tserver with Master: 391d2c8b44ee4535bcef7dc62643e6c0 (127.4.140.195:46131)
I20260928 02:20:37.582660 17534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:36635
I20260928 02:20:37.588727 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:20:37.607510 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:37.592257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.4.140.254:45447)
0928 02:20:37.592445 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:37.592453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:37.593231 (+ 778us) server_negotiation.cc:407] Connection header received
0928 02:20:37.593371 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:37.593376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:37.593474 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:37.593604 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:37.595014 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.596279 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.597661 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.598358 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.598495 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:37.601156 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:37.601163 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:37.601167 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:37.601207 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:37.604746 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:37.605397 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:37.605403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:37.605407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:37.605577 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:37.605941 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:37.605947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:37.605950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:37.606341 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:37.606605 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:37.606891 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 02:20:37.607155 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:20:37.612660 17534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:20:37.620273 17534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:20:37.638499 17876 heartbeater.cc:503] Master 127.4.140.254:45447 was elected leader, sending a full tablet report...
I20260928 02:20:37.657006 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:37.634819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:41911 (local address 127.0.0.1:60666)
0928 02:20:37.646097 (+ 11278us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:37.646145 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:37.646343 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:37.649128 (+ 2785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:37.649136 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:37.649168 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:20:37.649626 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:37.649637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.651583 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.651589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:37.653899 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:37.653914 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.656008 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.656014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:37.656224 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:37.656275 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 02:20:37.656401 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 02:20:37.656577 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10957,"mutex_wait_us":1,"thread_start_us":11706,"threads_started":1}
I20260928 02:20:37.658470 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:37.635396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.4.140.194:41911)
0928 02:20:37.648269 (+ 12873us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:37.648278 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:37.648306 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:20:37.648404 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:37.648412 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:37.648672 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:37.648858 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:20:37.649829 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.651415 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.654085 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.655489 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.655615 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:37.655837 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:37.657974 (+ 2137us) server_negotiation.cc:299] Negotiation successful
0928 02:20:37.658159 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12703,"thread_start_us":12546,"threads_started":1}
I20260928 02:20:37.661270 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:37.635129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:44869 (local address 127.0.0.1:34500)
0928 02:20:37.650467 (+ 15338us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:37.650487 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:37.650603 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:37.651664 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:37.651672 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:37.651713 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:20:37.652261 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:37.652277 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.654810 (+ 2533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.654818 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:37.657548 (+ 2730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:37.657566 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.660245 (+ 2679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.660254 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:37.660486 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:37.660537 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 02:20:37.660624 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:20:37.660798 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15213,"thread_start_us":15072,"threads_started":1}
I20260928 02:20:37.662308 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:37.635835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34500 (local address 127.4.140.193:44869)
0928 02:20:37.647781 (+ 11946us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:37.647790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:37.650736 (+ 2946us) server_negotiation.cc:407] Connection header received
0928 02:20:37.650865 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:37.650875 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:37.651117 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:37.651343 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:20:37.652540 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.654592 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.657760 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.659622 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.659767 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:37.660077 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:37.661644 (+ 1567us) server_negotiation.cc:299] Negotiation successful
0928 02:20:37.661858 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11772,"thread_start_us":11609,"threads_started":1}
I20260928 02:20:37.662581 17811 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd9addcd67246a58151df91b1586c13 (DEFAULT_TABLE table=test-table [id=a27b0d03208a492cb4919b617283cc5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:20:37.664111 17811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd9addcd67246a58151df91b1586c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:20:37.667974 17666 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd9addcd67246a58151df91b1586c13 (DEFAULT_TABLE table=test-table [id=a27b0d03208a492cb4919b617283cc5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:20:37.669497 17666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd9addcd67246a58151df91b1586c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:20:37.670984 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:37.656927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:46131 (local address 127.0.0.1:41736)
0928 02:20:37.657732 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:37.657748 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:37.657829 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:37.664423 (+ 6594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:37.664429 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:37.664461 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:20:37.664662 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:37.664673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.666468 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.666473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:37.668198 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:37.668213 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.670205 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.670211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:37.670425 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:37.670465 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:20:37.670549 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 02:20:37.670690 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":6043,"threads_started":1}
I20260928 02:20:37.671701 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:37.651043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41736 (local address 127.4.140.195:46131)
0928 02:20:37.663514 (+ 12471us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:37.663522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:37.663550 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:20:37.663641 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:37.663648 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:37.663873 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:37.664107 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:20:37.664851 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.666309 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.668397 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.669766 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.669885 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:37.670107 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:37.671193 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0928 02:20:37.671371 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12313,"thread_start_us":12249,"threads_started":1}
I20260928 02:20:37.676724 17945 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd9addcd67246a58151df91b1586c13 (DEFAULT_TABLE table=test-table [id=a27b0d03208a492cb4919b617283cc5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:20:37.678588 17945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd9addcd67246a58151df91b1586c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:20:37.687374 18039 tablet_bootstrap.cc:492] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447: Bootstrap starting.
I20260928 02:20:37.691608 18038 tablet_bootstrap.cc:492] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563: Bootstrap starting.
I20260928 02:20:37.693240 18039 tablet_bootstrap.cc:654] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447: Neither blocks nor log segments found. Creating new log.
I20260928 02:20:37.694615 18039 log.cc:824] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447: Log is configured to *not* fsync() on all Append() calls
I20260928 02:20:37.695650 18038 tablet_bootstrap.cc:654] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563: Neither blocks nor log segments found. Creating new log.
I20260928 02:20:37.696947 18038 log.cc:824] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563: Log is configured to *not* fsync() on all Append() calls
I20260928 02:20:37.703507 18038 tablet_bootstrap.cc:492] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563: No bootstrap required, opened a new log
I20260928 02:20:37.703876 18038 ts_tablet_manager.cc:1397] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260928 02:20:37.710733 18042 tablet_bootstrap.cc:492] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0: Bootstrap starting.
I20260928 02:20:37.712586 18039 tablet_bootstrap.cc:492] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447: No bootstrap required, opened a new log
I20260928 02:20:37.712867 18039 ts_tablet_manager.cc:1397] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.017s
I20260928 02:20:37.714596 18042 tablet_bootstrap.cc:654] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0: Neither blocks nor log segments found. Creating new log.
I20260928 02:20:37.715970 18042 log.cc:824] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:20:37.719075 18038 raft_consensus.cc:348] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3755de48ad248f9a814abe9e1eaa563" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41911 } } peers { permanent_uuid: "da0ff51095e243de80f05b608a903447" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44869 } } peers { permanent_uuid: "391d2c8b44ee4535bcef7dc62643e6c0" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46131 } }
I20260928 02:20:37.719694 18038 raft_consensus.cc:374] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:20:37.719810 18038 raft_consensus.cc:729] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3755de48ad248f9a814abe9e1eaa563, State: Initialized, Role: FOLLOWER
I20260928 02:20:37.720757 18038 consensus_queue.cc:260] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3755de48ad248f9a814abe9e1eaa563" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41911 } } peers { permanent_uuid: "da0ff51095e243de80f05b608a903447" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44869 } } peers { permanent_uuid: "391d2c8b44ee4535bcef7dc62643e6c0" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46131 } }
I20260928 02:20:37.723416 18039 raft_consensus.cc:348] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3755de48ad248f9a814abe9e1eaa563" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41911 } } peers { permanent_uuid: "da0ff51095e243de80f05b608a903447" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44869 } } peers { permanent_uuid: "391d2c8b44ee4535bcef7dc62643e6c0" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46131 } }
I20260928 02:20:37.723927 18039 raft_consensus.cc:374] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:20:37.724072 18039 raft_consensus.cc:729] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da0ff51095e243de80f05b608a903447, State: Initialized, Role: FOLLOWER
I20260928 02:20:37.724802 18039 consensus_queue.cc:260] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3755de48ad248f9a814abe9e1eaa563" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41911 } } peers { permanent_uuid: "da0ff51095e243de80f05b608a903447" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44869 } } peers { permanent_uuid: "391d2c8b44ee4535bcef7dc62643e6c0" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46131 } }
I20260928 02:20:37.727257 17731 heartbeater.cc:503] Master 127.4.140.254:45447 was elected leader, sending a full tablet report...
I20260928 02:20:37.729475 18038 ts_tablet_manager.cc:1428] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563: Time spent starting tablet: real 0.025s user 0.010s sys 0.008s
I20260928 02:20:37.730862 18039 ts_tablet_manager.cc:1428] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447: Time spent starting tablet: real 0.018s user 0.003s sys 0.010s
I20260928 02:20:37.732153 18042 tablet_bootstrap.cc:492] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0: No bootstrap required, opened a new log
I20260928 02:20:37.732448 18042 ts_tablet_manager.cc:1397] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.004s
I20260928 02:20:37.739619 18042 raft_consensus.cc:348] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3755de48ad248f9a814abe9e1eaa563" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41911 } } peers { permanent_uuid: "da0ff51095e243de80f05b608a903447" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44869 } } peers { permanent_uuid: "391d2c8b44ee4535bcef7dc62643e6c0" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46131 } }
I20260928 02:20:37.740027 18042 raft_consensus.cc:374] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:20:37.740147 18042 raft_consensus.cc:729] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391d2c8b44ee4535bcef7dc62643e6c0, State: Initialized, Role: FOLLOWER
I20260928 02:20:37.740845 18042 consensus_queue.cc:260] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3755de48ad248f9a814abe9e1eaa563" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41911 } } peers { permanent_uuid: "da0ff51095e243de80f05b608a903447" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44869 } } peers { permanent_uuid: "391d2c8b44ee4535bcef7dc62643e6c0" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46131 } }
I20260928 02:20:37.742048 18017 heartbeater.cc:503] Master 127.4.140.254:45447 was elected leader, sending a full tablet report...
I20260928 02:20:37.742702 18042 ts_tablet_manager.cc:1428] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
W20260928 02:20:37.797173 18018 tablet.cc:2369] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:20:37.841007 17877 tablet.cc:2369] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:20:37.855540 18044 raft_consensus.cc:482] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:20:37.855860 18044 raft_consensus.cc:504] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3755de48ad248f9a814abe9e1eaa563" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41911 } } peers { permanent_uuid: "da0ff51095e243de80f05b608a903447" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44869 } } peers { permanent_uuid: "391d2c8b44ee4535bcef7dc62643e6c0" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46131 } }
I20260928 02:20:37.857657 18044 leader_election.cc:288] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da0ff51095e243de80f05b608a903447 (127.4.140.193:44869), 391d2c8b44ee4535bcef7dc62643e6c0 (127.4.140.195:46131)
I20260928 02:20:37.864008 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:37.858059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:44085 (local address 127.4.140.193:44869)
0928 02:20:37.858338 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:37.858348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:37.858790 (+ 442us) server_negotiation.cc:407] Connection header received
0928 02:20:37.859021 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:37.859029 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:37.859123 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:37.859219 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:20:37.859837 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.860600 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.862310 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.863072 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.863245 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:37.863323 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:37.863601 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 02:20:37.863735 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:20:37.864074 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:37.857987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:44869 (local address 127.4.140.194:44085)
0928 02:20:37.858658 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:37.858674 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:37.858872 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:37.859322 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:37.859326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:37.859350 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:20:37.859668 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:37.859676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.860730 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.860736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:37.862146 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:37.862157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.863288 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.863294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:37.863444 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:37.863467 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:20:37.863574 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:20:37.863700 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":363,"threads_started":1}
I20260928 02:20:37.864511 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:37.859015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:54977 (local address 127.4.140.195:46131)
0928 02:20:37.859213 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:37.859219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:37.859530 (+ 311us) server_negotiation.cc:407] Connection header received
0928 02:20:37.859736 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:37.859740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:37.859800 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:37.859895 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:20:37.860410 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.861128 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.862757 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.863519 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.863706 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:37.863780 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:37.864127 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 02:20:37.864260 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:20:37.864569 18048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:37.858793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:46131 (local address 127.4.140.194:54977)
0928 02:20:37.859423 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:37.859439 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:37.859583 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:37.859933 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:37.859937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:37.859960 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:20:37.860270 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:37.860277 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.861281 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.861286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:37.862607 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:37.862619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.863739 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.863754 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:37.863963 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:37.863996 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:20:37.864102 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:20:37.864227 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":314,"threads_started":1}
I20260928 02:20:37.865156 17686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd9addcd67246a58151df91b1586c13" candidate_uuid: "f3755de48ad248f9a814abe9e1eaa563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da0ff51095e243de80f05b608a903447" is_pre_election: true
I20260928 02:20:37.865655 17972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd9addcd67246a58151df91b1586c13" candidate_uuid: "f3755de48ad248f9a814abe9e1eaa563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391d2c8b44ee4535bcef7dc62643e6c0" is_pre_election: true
I20260928 02:20:37.865787 17686 raft_consensus.cc:2454] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3755de48ad248f9a814abe9e1eaa563 in term 0.
I20260928 02:20:37.866178 17972 raft_consensus.cc:2454] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3755de48ad248f9a814abe9e1eaa563 in term 0.
I20260928 02:20:37.866565 17764 leader_election.cc:302] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da0ff51095e243de80f05b608a903447, f3755de48ad248f9a814abe9e1eaa563; no voters:
I20260928 02:20:37.866933 18044 raft_consensus.cc:2781] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:20:37.867151 18044 raft_consensus.cc:482] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:20:37.867280 18044 raft_consensus.cc:3037] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:20:37.869635 18044 raft_consensus.cc:504] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3755de48ad248f9a814abe9e1eaa563" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41911 } } peers { permanent_uuid: "da0ff51095e243de80f05b608a903447" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44869 } } peers { permanent_uuid: "391d2c8b44ee4535bcef7dc62643e6c0" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46131 } }
I20260928 02:20:37.870138 18044 leader_election.cc:288] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [CANDIDATE]: Term 1 election: Requested vote from peers da0ff51095e243de80f05b608a903447 (127.4.140.193:44869), 391d2c8b44ee4535bcef7dc62643e6c0 (127.4.140.195:46131)
I20260928 02:20:37.870566 17686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd9addcd67246a58151df91b1586c13" candidate_uuid: "f3755de48ad248f9a814abe9e1eaa563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da0ff51095e243de80f05b608a903447"
I20260928 02:20:37.870640 17972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd9addcd67246a58151df91b1586c13" candidate_uuid: "f3755de48ad248f9a814abe9e1eaa563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391d2c8b44ee4535bcef7dc62643e6c0"
I20260928 02:20:37.870771 17686 raft_consensus.cc:3037] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:20:37.870893 17972 raft_consensus.cc:3037] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:20:37.873311 17972 raft_consensus.cc:2454] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3755de48ad248f9a814abe9e1eaa563 in term 1.
I20260928 02:20:37.873514 17686 raft_consensus.cc:2454] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3755de48ad248f9a814abe9e1eaa563 in term 1.
I20260928 02:20:37.873687 17764 leader_election.cc:302] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391d2c8b44ee4535bcef7dc62643e6c0, f3755de48ad248f9a814abe9e1eaa563; no voters:
I20260928 02:20:37.874006 18044 raft_consensus.cc:2781] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:20:37.874500 18044 raft_consensus.cc:686] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [term 1 LEADER]: Becoming Leader. State: Replica: f3755de48ad248f9a814abe9e1eaa563, State: Running, Role: LEADER
I20260928 02:20:37.874974 18044 consensus_queue.cc:237] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3755de48ad248f9a814abe9e1eaa563" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41911 } } peers { permanent_uuid: "da0ff51095e243de80f05b608a903447" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44869 } } peers { permanent_uuid: "391d2c8b44ee4535bcef7dc62643e6c0" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46131 } }
I20260928 02:20:37.879873 17534 catalog_manager.cc:5495] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3755de48ad248f9a814abe9e1eaa563 (127.4.140.194). New cstate: current_term: 1 leader_uuid: "f3755de48ad248f9a814abe9e1eaa563" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3755de48ad248f9a814abe9e1eaa563" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 41911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da0ff51095e243de80f05b608a903447" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "391d2c8b44ee4535bcef7dc62643e6c0" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46131 } health_report { overall_health: UNKNOWN } } }
W20260928 02:20:37.916379 17732 tablet.cc:2369] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:20:37.926342 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:37.921114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.4.140.194:41911)
0928 02:20:37.921319 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:37.921324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:37.921341 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:20:37.921518 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:37.921522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:37.921599 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:37.921720 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:20:37.922038 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.922788 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.924246 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.924636 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.924727 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:37.925189 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:20:37.925715 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:20:37.925899 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:37.925987 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 02:20:37.926068 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:20:37.934527 17686 raft_consensus.cc:1261] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447 [term 1 FOLLOWER]: Refusing update from remote peer f3755de48ad248f9a814abe9e1eaa563: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:20:37.934651 17972 raft_consensus.cc:1261] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0 [term 1 FOLLOWER]: Refusing update from remote peer f3755de48ad248f9a814abe9e1eaa563: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:20:37.935575 18044 consensus_queue.cc:1035] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da0ff51095e243de80f05b608a903447" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:20:37.935881 18049 consensus_queue.cc:1035] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [LEADER]: Connected to new peer: Peer: permanent_uuid: "391d2c8b44ee4535bcef7dc62643e6c0" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 46131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:20:37.947643 18052 mvcc.cc:205] Tried to move back new op lower bound from 7334142107369656320 to 7334142107139895296. Current Snapshot: MvccSnapshot[applied={T|T < 7334142107369656320}]
I20260928 02:20:37.962908 17534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:20:37.963191 17534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:20:37.966934 17534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16e21b80676f4b398843b3d3755f7021: Sending DeleteTablet for 3 replicas of tablet 0bd9addcd67246a58151df91b1586c13
I20260928 02:20:37.968202 17811 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bd9addcd67246a58151df91b1586c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:20:37 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:60666
I20260928 02:20:37.968351 17666 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bd9addcd67246a58151df91b1586c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:20:37 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:34500
I20260928 02:20:37.969111 18063 tablet_replica.cc:330] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563: stopping tablet replica
I20260928 02:20:37.969213 17945 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bd9addcd67246a58151df91b1586c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:20:37 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:41736
I20260928 02:20:37.969444 18063 raft_consensus.cc:2229] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:20:37.970047 18065 tablet_replica.cc:330] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0: stopping tablet replica
I20260928 02:20:37.970089 18063 raft_consensus.cc:2258] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:20:37.970453 18065 raft_consensus.cc:2229] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:20:37.970942 18065 raft_consensus.cc:2258] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:20:37.972661 18063 ts_tablet_manager.cc:1905] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:20:37.973006 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 17571
I20260928 02:20:37.973034 18065 ts_tablet_manager.cc:1905] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:20:37.980180 18065 ts_tablet_manager.cc:1918] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:20:37.980305 18063 ts_tablet_manager.cc:1918] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:20:37.980386 18065 log.cc:1196] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/0bd9addcd67246a58151df91b1586c13
I20260928 02:20:37.980491 18063 log.cc:1196] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/0bd9addcd67246a58151df91b1586c13
I20260928 02:20:37.980971 18065 ts_tablet_manager.cc:1939] T 0bd9addcd67246a58151df91b1586c13 P 391d2c8b44ee4535bcef7dc62643e6c0: Deleting consensus metadata
I20260928 02:20:37.981017 18063 ts_tablet_manager.cc:1939] T 0bd9addcd67246a58151df91b1586c13 P f3755de48ad248f9a814abe9e1eaa563: Deleting consensus metadata
I20260928 02:20:37.982741 17518 catalog_manager.cc:4841] TS 391d2c8b44ee4535bcef7dc62643e6c0 (127.4.140.195:46131): tablet 0bd9addcd67246a58151df91b1586c13 (table test-table [id=a27b0d03208a492cb4919b617283cc5f]) successfully deleted
I20260928 02:20:37.983215 17519 catalog_manager.cc:4841] TS f3755de48ad248f9a814abe9e1eaa563 (127.4.140.194:41911): tablet 0bd9addcd67246a58151df91b1586c13 (table test-table [id=a27b0d03208a492cb4919b617283cc5f]) successfully deleted
I20260928 02:20:37.984412 18064 tablet_replica.cc:330] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447: stopping tablet replica
I20260928 02:20:37.984810 18064 raft_consensus.cc:2229] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:20:37.985354 18064 raft_consensus.cc:2258] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:20:37.987363 18064 ts_tablet_manager.cc:1905] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:20:37.994181 18064 ts_tablet_manager.cc:1918] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:20:37.994362 18064 log.cc:1196] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/0bd9addcd67246a58151df91b1586c13
I20260928 02:20:37.994936 18064 ts_tablet_manager.cc:1939] T 0bd9addcd67246a58151df91b1586c13 P da0ff51095e243de80f05b608a903447: Deleting consensus metadata
I20260928 02:20:37.996903 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:37.974428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34516 (local address 127.4.140.193:44869)
0928 02:20:37.975005 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:37.975011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:37.975032 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:20:37.975092 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:37.975098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:37.975177 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:37.975304 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:37.981969 (+ 6665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.983136 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.985097 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:37.985754 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:37.985887 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:37.989339 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:37.989353 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:37.989371 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:37.989413 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:37.993595 (+ 4182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:37.994430 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:37.994441 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:37.994446 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:37.994623 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:37.995104 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:37.995112 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:37.995119 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:37.995578 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:37.995930 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:37.996269 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 02:20:37.996502 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447}
I20260928 02:20:37.996963 17518 catalog_manager.cc:4841] TS da0ff51095e243de80f05b608a903447 (127.4.140.193:44869): tablet 0bd9addcd67246a58151df91b1586c13 (table test-table [id=a27b0d03208a492cb4919b617283cc5f]) successfully deleted
I20260928 02:20:37.997591 17726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:20:38.089180 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 17571
I20260928 02:20:38.113343 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 17735
I20260928 02:20:38.131320 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:38.114289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60674 (local address 127.4.140.194:41911)
0928 02:20:38.114465 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:38.114471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:38.114557 (+ 86us) server_negotiation.cc:407] Connection header received
0928 02:20:38.114792 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:38.114799 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:38.114878 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:38.114984 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:38.116936 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:38.118210 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:38.120314 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:38.120995 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:38.121131 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:38.124912 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:38.124934 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:38.124959 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:38.125006 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:38.128422 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:38.129320 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:38.129328 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:38.129330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:38.129477 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:38.129919 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:38.129922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:38.129924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:38.130261 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:38.130483 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:38.130844 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 02:20:38.131023 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:20:38.132150 17871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:20:38.245319 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 17735
I20260928 02:20:38.263768 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 17878
I20260928 02:20:38.277415 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:38.264622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.4.140.195:46131)
0928 02:20:38.264801 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:38.264809 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:38.264870 (+ 61us) server_negotiation.cc:407] Connection header received
0928 02:20:38.265116 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:38.265120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:38.265184 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:38.265309 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:38.266678 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:38.267382 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:38.268852 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:38.269256 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:38.269391 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:38.271945 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:38.271988 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:38.272009 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:38.272053 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:38.274531 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:38.275209 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:38.275231 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:38.275233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:38.275367 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:38.275738 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:38.275744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:38.275750 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:38.276216 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:38.276553 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:38.276834 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 02:20:38.277045 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:20:38.278082 18012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:20:38.366753 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 17878
I20260928 02:20:38.383292 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 17501
I20260928 02:20:38.384531 17564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:20:38.496596 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 17501
I20260928 02:20:38.523192 4659 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 02:20:38.858718 4659 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 02:20:38.858944 4659 mini_ranger.cc:66] Stopping Ranger...
I20260928 02:20:39.204779 4659 mini_ranger.cc:68] Stopped Ranger
2026-09-28 02:20:39.205 UTC [16552] LOG: received smart shutdown request
2026-09-28 02:20:39.208 UTC [16552] LOG: background worker "logical replication launcher" (PID 16561) exited with exit code 1
2026-09-28 02:20:39.208 UTC [16556] LOG: shutting down
2026-09-28 02:20:39.399 UTC [16552] LOG: database system is shut down
2026-09-28T02:20:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54218 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:20:39 dist-test-slave-h37r krb5kdc[18078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:20:39 dist-test-slave-h37r krb5kdc[18078](info): set up 2 sockets
Sep 28 02:20:39 dist-test-slave-h37r krb5kdc[18078](info): commencing operation
krb5kdc: starting...
W20260928 02:20:40.648459 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:20:40 dist-test-slave-h37r krb5kdc[18078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:20:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:20:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:20:40.880280 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:42559
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:42559,127.4.140.253:38657,127.4.140.252:33581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:20:41.215431 18091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:41.215724 18091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:41.215813 18091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:41.222988 18091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:20:41.223088 18091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:41.223137 18091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:41.223194 18091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:20:41.223239 18091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:20:41.233184 18091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:42559,127.4.140.253:38657,127.4.140.252:33581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:42559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:41.234431 18091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:41.235831 18091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:41.247782 18096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:41.247821 18097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:41.247745 18099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:41.248436 18091 server_base.cc:1034] running on GCE node
I20260928 02:20:41.249178 18091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:41.250555 18091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:41.251780 18091 hybrid_clock.cc:648] HybridClock initialized: now 1790562041251742 us; error 50 us; skew 500 ppm
I20260928 02:20:41.252128 18091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:41 dist-test-slave-h37r krb5kdc[18078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562041, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:41.260082 18091 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:20:41.261564 18091 webserver.cc:460] Webserver started at http://127.4.140.254:37845/ using document root <none> and password file <none>
I20260928 02:20:41.262247 18091 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:41.262391 18091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:20:41.262689 18091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:20:41.265687 18091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "f1c67ae843784da8853f9508be04d665"
format_stamp: "Formatted at 2026-09-28 02:20:41 on dist-test-slave-h37r"
server_key: "b438c4ad646d2bb87291681e06035dcd"
server_key_iv: "646aa7b8601628a27d1e9637474d2163"
server_key_version: "encryptionkey@0"
I20260928 02:20:41.266491 18091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "f1c67ae843784da8853f9508be04d665"
format_stamp: "Formatted at 2026-09-28 02:20:41 on dist-test-slave-h37r"
server_key: "b438c4ad646d2bb87291681e06035dcd"
server_key_iv: "646aa7b8601628a27d1e9637474d2163"
server_key_version: "encryptionkey@0"
I20260928 02:20:41.271617 18091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260928 02:20:41.275004 18106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:41.276311 18091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:20:41.276489 18091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "f1c67ae843784da8853f9508be04d665"
format_stamp: "Formatted at 2026-09-28 02:20:41 on dist-test-slave-h37r"
server_key: "b438c4ad646d2bb87291681e06035dcd"
server_key_iv: "646aa7b8601628a27d1e9637474d2163"
server_key_version: "encryptionkey@0"
I20260928 02:20:41.276678 18091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:41.292778 18091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:41.296828 18091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:41.297117 18091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:41.315632 18091 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:42559
I20260928 02:20:41.315670 18157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:42559 every 8 connection(s)
I20260928 02:20:41.317032 18091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:20:41.320748 18158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:20:41.322968 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 18091
I20260928 02:20:41.323184 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:20:41.324026 4659 external_mini_cluster.cc:1428] Setting key 9e12ee874e47019258bb42342c2977e7
I20260928 02:20:41.330261 18158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:20:41 dist-test-slave-h37r krb5kdc[18078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562041, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.140.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.253@KRBTEST.COM" created.
I20260928 02:20:41.354828 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.334172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43944 (local address 127.4.140.254:42559)
0928 02:20:41.335287 (+ 1115us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:41.335304 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:41.335724 (+ 420us) server_negotiation.cc:407] Connection header received
0928 02:20:41.337794 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:41.337804 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:41.338130 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:41.338369 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:41.339513 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:41.341498 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:41.343080 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:41.344141 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:41.344367 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:41.347530 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:41.347538 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:41.347550 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:41.347578 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:41.350608 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:41.351507 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:41.351512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:41.351515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:41.351652 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:41.352146 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:41.352149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:41.352151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:41.352616 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:41.352764 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:41.353464 (+ 700us) server_negotiation.cc:299] Negotiation successful
0928 02:20:41.353772 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":823,"thread_start_us":410,"threads_started":1}
I20260928 02:20:41.354863 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.334172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:42559 (local address 127.0.0.1:43944)
0928 02:20:41.335529 (+ 1357us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:41.335583 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:41.337405 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:41.338396 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:41.338401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:41.338962 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:41.339311 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:41.339329 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:41.341740 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:41.341744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:41.342904 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:41.342921 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:41.344369 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:41.344374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:41.344489 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:41.345185 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:41.345207 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:41.347321 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:41.350742 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:41.350756 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:41.350770 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:41.351381 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:41.351786 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:41.351794 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:41.351801 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:41.352036 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:41.352780 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:41.352791 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:41.353071 (+ 280us) client_negotiation.cc:769] Sending connection context
0928 02:20:41.353373 (+ 302us) client_negotiation.cc:240] Negotiation successful
0928 02:20:41.353976 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1087,"mutex_wait_us":84,"spinlock_wait_cycles":25984,"thread_start_us":809,"threads_started":1}
I20260928 02:20:41.357877 18158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:20:41.358903 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.358503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38657 (local address 127.0.0.1:49486)
0928 02:20:41.358665 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:41.358788 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 02:20:41.359146 18110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38657: connect: Connection refused (error 111)
I20260928 02:20:41.360795 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.360444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38657 (local address 127.0.0.1:49488)
0928 02:20:41.360598 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:41.360661 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 02:20:41.360983 18158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.253:38657: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38657: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.253.keytab.
Entry for principal kudu/127.4.140.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.253.keytab.
I20260928 02:20:41.378999 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.253:38657
--webserver_interface=127.4.140.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.253.keytab
--principal=kudu/127.4.140.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:42559,127.4.140.253:38657,127.4.140.252:33581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:20:41.410331 18158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } attempt: 1
I20260928 02:20:41.411620 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.411088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38657 (local address 127.0.0.1:49492)
0928 02:20:41.411310 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:41.411453 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 02:20:41.412665 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.412275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38657 (local address 127.0.0.1:49498)
0928 02:20:41.412423 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:41.412508 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:20:41.412933 18158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.253:38657: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38657: connect: Connection refused (error 111)
I20260928 02:20:41.481213 18158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } attempt: 2
I20260928 02:20:41.482549 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.481912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38657 (local address 127.0.0.1:49510)
0928 02:20:41.482196 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:41.482334 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 02:20:41.483695 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.483277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38657 (local address 127.0.0.1:49516)
0928 02:20:41.483445 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:41.483533 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:20:41.483943 18158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.253:38657: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38657: connect: Connection refused (error 111)
I20260928 02:20:41.575268 18158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } attempt: 3
I20260928 02:20:41.576535 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.576014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38657 (local address 127.0.0.1:49530)
0928 02:20:41.576243 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:41.576362 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 02:20:41.577728 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.577296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38657 (local address 127.0.0.1:49540)
0928 02:20:41.577459 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:41.577563 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 02:20:41.577935 18158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.253:38657: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38657: connect: Connection refused (error 111)
W20260928 02:20:41.709045 18165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:41.709343 18165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:41.709434 18165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:41.716706 18165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:20:41.716821 18165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:41.716867 18165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:41.716924 18165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:20:41.716961 18165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:20:41.721197 18158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } attempt: 4
I20260928 02:20:41.722508 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.721885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38657 (local address 127.0.0.1:49552)
0928 02:20:41.722207 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:41.722331 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260928 02:20:41.723557 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.723150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38657 (local address 127.0.0.1:49556)
0928 02:20:41.723308 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:41.723405 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 02:20:41.723763 18158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.253:38657: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38657: connect: Connection refused (error 111)
I20260928 02:20:41.727852 18165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:42559,127.4.140.253:38657,127.4.140.252:33581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.253.keytab
--principal=kudu/127.4.140.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.253:38657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.140.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:41.729290 18165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:41.730607 18165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:41.742687 18171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:41.742738 18173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:41.742794 18170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:41.748869 18165 server_base.cc:1034] running on GCE node
I20260928 02:20:41.749728 18165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:41.751294 18165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:41.752534 18165 hybrid_clock.cc:648] HybridClock initialized: now 1790562041752488 us; error 59 us; skew 500 ppm
I20260928 02:20:41.752897 18165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:41 dist-test-slave-h37r krb5kdc[18078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562041, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:41.756155 18165 init.cc:377] Logged in from keytab as kudu/127.4.140.253@KRBTEST.COM (short username kudu)
I20260928 02:20:41.757696 18165 webserver.cc:460] Webserver started at http://127.4.140.253:39815/ using document root <none> and password file <none>
I20260928 02:20:41.758435 18165 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:41.758584 18165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:20:41.758893 18165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:20:41.761880 18165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-1/data/instance:
uuid: "aa007486d16a452e9e12ee39531142d5"
format_stamp: "Formatted at 2026-09-28 02:20:41 on dist-test-slave-h37r"
server_key: "7e10832b8176cab9e444f6a9165edba5"
server_key_iv: "ef358309eb25e1faf91cb8f5b43c8777"
server_key_version: "encryptionkey@0"
I20260928 02:20:41.762681 18165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-1/wal/instance:
uuid: "aa007486d16a452e9e12ee39531142d5"
format_stamp: "Formatted at 2026-09-28 02:20:41 on dist-test-slave-h37r"
server_key: "7e10832b8176cab9e444f6a9165edba5"
server_key_iv: "ef358309eb25e1faf91cb8f5b43c8777"
server_key_version: "encryptionkey@0"
I20260928 02:20:41.767772 18165 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:20:41.771117 18180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:41.772475 18165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:20:41.772661 18165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-1/wal
uuid: "aa007486d16a452e9e12ee39531142d5"
format_stamp: "Formatted at 2026-09-28 02:20:41 on dist-test-slave-h37r"
server_key: "7e10832b8176cab9e444f6a9165edba5"
server_key_iv: "ef358309eb25e1faf91cb8f5b43c8777"
server_key_version: "encryptionkey@0"
I20260928 02:20:41.772869 18165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:41.792652 18165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:41.796757 18165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:41.797050 18165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:41.815946 18165 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.253:38657
I20260928 02:20:41.815989 18231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.253:38657 every 8 connection(s)
I20260928 02:20:41.817414 18165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-1/data/info.pb
I20260928 02:20:41.820956 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 18165
I20260928 02:20:41.821187 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-1/wal/instance
I20260928 02:20:41.821564 18232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:20:41.821779 4659 external_mini_cluster.cc:1428] Setting key 543aa901ab5ce093ce6edc833c74f18f
I20260928 02:20:41.831663 18232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.140.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.252@KRBTEST.COM" created.
Sep 28 02:20:41 dist-test-slave-h37r krb5kdc[18078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562041, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.253@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:20:41.854569 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.834627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43958 (local address 127.4.140.254:42559)
0928 02:20:41.834816 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:41.834820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:41.836463 (+ 1643us) server_negotiation.cc:407] Connection header received
0928 02:20:41.838439 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:41.838445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:41.838541 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:41.838653 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:41.840419 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:41.841131 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:41.842983 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:41.843408 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:41.843549 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:41.848268 (+ 4719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:41.848273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:41.848276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:41.848310 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:41.851005 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:41.851997 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:41.852001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:41.852004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:41.852121 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:41.852607 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:41.852610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:41.852612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:41.852884 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:41.852996 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:41.853898 (+ 902us) server_negotiation.cc:299] Negotiation successful
0928 02:20:41.854159 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:20:41.855260 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.835200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:42559 (local address 127.0.0.1:43958)
0928 02:20:41.836252 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:41.836320 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:41.838128 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:41.838912 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:41.838921 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:41.839466 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:41.840240 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:41.840260 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:41.841306 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:41.841312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:41.842838 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:41.842850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:41.843641 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:41.843647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:41.843953 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:41.844591 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:41.844614 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:41.848017 (+ 3403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:41.851173 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:41.851185 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:41.851205 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:41.851856 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:41.852236 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:41.852244 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:41.852250 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:41.852491 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:41.853015 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:41.853026 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:41.853494 (+ 468us) client_negotiation.cc:769] Sending connection context
0928 02:20:41.853722 (+ 228us) client_negotiation.cc:240] Negotiation successful
0928 02:20:41.854104 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":412,"threads_started":1}
I20260928 02:20:41.857075 18232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:20:41 dist-test-slave-h37r krb5kdc[18078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562041, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.253@KRBTEST.COM for kudu/127.4.140.253@KRBTEST.COM
I20260928 02:20:41.871963 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.857768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38657 (local address 127.0.0.1:49560)
0928 02:20:41.857919 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:41.857938 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:41.858032 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:41.859131 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:41.859135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:41.859586 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:41.859708 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:41.859716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:41.861103 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:41.861111 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:41.861926 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:41.861936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:41.862811 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:41.862816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:41.863033 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:41.863635 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:41.863651 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:41.865878 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:41.869154 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:41.869162 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:41.869166 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:41.869570 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:41.869975 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:41.869982 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:41.869985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:41.870159 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:41.870834 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:41.870838 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:41.870982 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 02:20:41.871121 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 02:20:41.871478 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 02:20:41.872002 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.858080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49560 (local address 127.4.140.253:38657)
0928 02:20:41.858736 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:41.858741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:41.858766 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:20:41.858808 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:41.858814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:41.858986 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:41.859145 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:41.859859 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:41.860968 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:41.862057 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:41.862635 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:41.862804 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:41.866082 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:41.866093 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:41.866107 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:41.866138 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:41.869033 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:41.869715 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:41.869723 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:41.869726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:41.869834 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:41.870269 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:41.870275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:41.870278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:41.870669 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:41.870888 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:41.871274 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 02:20:41.871596 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":508,"thread_start_us":371,"threads_started":1}
I20260928 02:20:41.874615 18232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.252.keytab.
Entry for principal kudu/127.4.140.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.252.keytab.
I20260928 02:20:41.875790 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.875346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51692)
0928 02:20:41.875505 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:41.875631 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:20:41.876072 18181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
I20260928 02:20:41.876566 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.252:33581
--webserver_interface=127.4.140.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.252.keytab
--principal=kudu/127.4.140.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:42559,127.4.140.253:38657,127.4.140.252:33581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:20:41.877720 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.877371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51706)
0928 02:20:41.877526 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:41.877624 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 02:20:41.877975 18232 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:33581: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
I20260928 02:20:41.927333 18232 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } attempt: 1
I20260928 02:20:41.928604 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.928155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51712)
0928 02:20:41.928350 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:41.928454 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 02:20:41.929610 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.929264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51728)
0928 02:20:41.929436 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:41.929502 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 02:20:41.929824 18232 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:33581: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
I20260928 02:20:41.998021 18232 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } attempt: 2
I20260928 02:20:41.999157 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.998695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51736)
0928 02:20:41.998875 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:41.998975 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 02:20:42.000351 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:41.999984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51740)
0928 02:20:42.000120 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:42.000203 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 02:20:42.000587 18232 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:33581: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
I20260928 02:20:42.023013 18158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } attempt: 5
Sep 28 02:20:42 dist-test-slave-h37r krb5kdc[18078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562041, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for kudu/127.4.140.253@KRBTEST.COM
I20260928 02:20:42.037568 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.023933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49566 (local address 127.4.140.253:38657)
0928 02:20:42.024119 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:42.024124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:42.024207 (+ 83us) server_negotiation.cc:407] Connection header received
0928 02:20:42.024416 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:42.024420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:42.024500 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:42.024585 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:42.025655 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.026373 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.028067 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.028473 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.028571 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:42.032166 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:42.032171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:42.032174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:42.032205 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:42.034828 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.035541 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.035545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.035547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.035664 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.036149 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.036153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.036155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.036434 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:42.036596 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:42.037011 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 02:20:42.037238 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:20:42.037568 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.023801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38657 (local address 127.0.0.1:49566)
0928 02:20:42.024089 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:42.024116 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:42.024288 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:42.024673 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:42.024680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:42.025232 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:42.025467 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:42.025480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.026513 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.026519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:42.027924 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:42.027941 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.028632 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.028638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:42.028829 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:42.029474 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:42.029504 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:42.031968 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:42.034984 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:42.034990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:42.034994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:42.035400 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:42.035799 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:42.035804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:42.035808 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:42.036017 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:42.036558 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:42.036563 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:42.036735 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 02:20:42.036883 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 02:20:42.037173 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 02:20:42.038657 18158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:20:42.039850 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.039462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51754)
0928 02:20:42.039630 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:42.039707 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 02:20:42.040904 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.040517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51766)
0928 02:20:42.040685 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:42.040770 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 02:20:42.041117 18158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:33581: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
I20260928 02:20:42.091840 18232 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } attempt: 3
I20260928 02:20:42.092350 18158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } attempt: 1
I20260928 02:20:42.093245 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.092678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51772)
0928 02:20:42.092945 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:42.093058 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260928 02:20:42.093457 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.092959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51782)
0928 02:20:42.093171 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:42.093273 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 02:20:42.094452 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.094069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51798)
0928 02:20:42.094225 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:42.094320 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 02:20:42.094588 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.094006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51786)
0928 02:20:42.094350 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:42.094438 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242}
W20260928 02:20:42.094818 18232 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:33581: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
W20260928 02:20:42.094864 18158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:33581: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
I20260928 02:20:42.163134 18158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } attempt: 2
I20260928 02:20:42.164357 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.163841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51804)
0928 02:20:42.164094 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:42.164191 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 02:20:42.165477 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.165150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51806)
0928 02:20:42.165285 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:42.165365 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 02:20:42.165704 18158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:33581: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
W20260928 02:20:42.227324 18239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:42.227612 18239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:42.227771 18239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:42.234951 18239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:20:42.235059 18239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:42.235103 18239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:42.235152 18239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:20:42.235234 18239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:20:42.238056 18232 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } attempt: 4
I20260928 02:20:42.239200 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.238740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51812)
0928 02:20:42.238956 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:42.239045 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 02:20:42.240218 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.239847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51824)
0928 02:20:42.240034 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:42.240111 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260928 02:20:42.240449 18232 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:33581: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
I20260928 02:20:42.245967 18239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:42559,127.4.140.253:38657,127.4.140.252:33581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.252.keytab
--principal=kudu/127.4.140.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.252:33581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.140.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:42.247399 18239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:42.248773 18239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:42.260828 18244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:42.260932 18245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:42.260839 18247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:42.262135 18239 server_base.cc:1034] running on GCE node
I20260928 02:20:42.262923 18239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:42.264420 18239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:42.265651 18239 hybrid_clock.cc:648] HybridClock initialized: now 1790562042265608 us; error 65 us; skew 500 ppm
I20260928 02:20:42.266014 18239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:42 dist-test-slave-h37r krb5kdc[18078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562042, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:42.269272 18239 init.cc:377] Logged in from keytab as kudu/127.4.140.252@KRBTEST.COM (short username kudu)
I20260928 02:20:42.270709 18239 webserver.cc:460] Webserver started at http://127.4.140.252:46113/ using document root <none> and password file <none>
I20260928 02:20:42.271368 18239 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:42.271463 18239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:20:42.271709 18239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:20:42.271945 18158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } attempt: 3
I20260928 02:20:42.273080 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.272619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51828)
0928 02:20:42.272841 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:42.272924 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 02:20:42.274214 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.273809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51838)
0928 02:20:42.274015 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:42.274096 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260928 02:20:42.274503 18158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:33581: Network error: Client connection negotiation failed: client connection to 127.4.140.252:33581: connect: Connection refused (error 111)
I20260928 02:20:42.275427 18239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-2/data/instance:
uuid: "ad3d3b3112034b7a89dba60fdc518582"
format_stamp: "Formatted at 2026-09-28 02:20:42 on dist-test-slave-h37r"
server_key: "13fba3d703919e50adcf9a53e0807e8f"
server_key_iv: "cdf580715a278366c926f02168f1cd07"
server_key_version: "encryptionkey@0"
I20260928 02:20:42.276376 18239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-2/wal/instance:
uuid: "ad3d3b3112034b7a89dba60fdc518582"
format_stamp: "Formatted at 2026-09-28 02:20:42 on dist-test-slave-h37r"
server_key: "13fba3d703919e50adcf9a53e0807e8f"
server_key_iv: "cdf580715a278366c926f02168f1cd07"
server_key_version: "encryptionkey@0"
I20260928 02:20:42.281807 18239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:20:42.285205 18254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:42.286523 18239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:20:42.286700 18239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-2/wal
uuid: "ad3d3b3112034b7a89dba60fdc518582"
format_stamp: "Formatted at 2026-09-28 02:20:42 on dist-test-slave-h37r"
server_key: "13fba3d703919e50adcf9a53e0807e8f"
server_key_iv: "cdf580715a278366c926f02168f1cd07"
server_key_version: "encryptionkey@0"
I20260928 02:20:42.286907 18239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:42.300621 18239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:42.304464 18239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:42.304754 18239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:42.323596 18239 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.252:33581
I20260928 02:20:42.323648 18305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.252:33581 every 8 connection(s)
I20260928 02:20:42.325007 18239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-2/data/info.pb
I20260928 02:20:42.328663 18306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:20:42.328667 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 18239
I20260928 02:20:42.328882 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561835360118-4659-0/minicluster-data/master-2/wal/instance
I20260928 02:20:42.329391 4659 external_mini_cluster.cc:1428] Setting key 39d189fd29bbb47a87e5b079caaa54a5
I20260928 02:20:42.338606 18306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:20:42 dist-test-slave-h37r krb5kdc[18078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.252@KRBTEST.COM
Sep 28 02:20:42 dist-test-slave-h37r krb5kdc[18078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562042, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.252@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:20:42.353116 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.331182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.4.140.252:33581)
0928 02:20:42.332085 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:42.332099 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:42.332156 (+ 57us) server_negotiation.cc:407] Connection header received
0928 02:20:42.333823 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:42.333856 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:42.334377 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:42.334833 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:42.336210 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.338140 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.339490 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.340491 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.340667 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:42.345336 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:42.345343 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:42.345357 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:42.345393 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:42.349098 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.349865 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.349874 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.349883 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.350068 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.350463 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.350466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.350468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.350957 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:42.351247 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:42.351705 (+ 458us) server_negotiation.cc:299] Negotiation successful
0928 02:20:42.352140 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":365,"threads_started":1}
I20260928 02:20:42.355984 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.341430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43966 (local address 127.4.140.254:42559)
0928 02:20:42.341676 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:42.341682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:42.342395 (+ 713us) server_negotiation.cc:407] Connection header received
0928 02:20:42.342574 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:42.342582 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:42.342670 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:42.342800 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:42.343935 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.344695 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.346184 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.346612 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.346709 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:42.350395 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:42.350404 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:42.350410 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:42.350455 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:42.353095 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.353948 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.353952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.353954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.354090 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.354480 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.354486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.354489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.354736 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:42.354904 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:42.355354 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 02:20:42.355566 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 02:20:42.355998 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.341433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:42559 (local address 127.0.0.1:43966)
0928 02:20:42.342241 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:42.342288 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:42.342435 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:42.342857 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:42.342861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:42.343398 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:42.343724 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:42.343752 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.344870 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.344876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:42.346046 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:42.346058 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.346814 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.346824 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:42.346974 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:42.347557 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:42.347581 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:42.350212 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:42.353254 (+ 3042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:42.353263 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:42.353269 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:42.353836 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:42.354210 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:42.354215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:42.354218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:42.354365 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:42.354870 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:42.354877 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:42.355048 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 02:20:42.355234 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 02:20:42.355534 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":440,"threads_started":1}
I20260928 02:20:42.357474 18306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:20:42 dist-test-slave-h37r krb5kdc[18078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562042, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.252@KRBTEST.COM for kudu/127.4.140.253@KRBTEST.COM
Sep 28 02:20:42 dist-test-slave-h37r krb5kdc[18078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:20:42.371037 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.358104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38657 (local address 127.0.0.1:49576)
0928 02:20:42.358242 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:42.358261 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:42.358376 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:42.358739 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:42.358747 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:42.359169 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:42.359340 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:42.359350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.360548 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.360552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:42.361560 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:42.361574 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.362494 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.362501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:42.362753 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:42.363238 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:42.363254 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:42.365328 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:42.368653 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:42.368662 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:42.368666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:42.369106 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:42.369481 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:42.369487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:42.369490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:42.369636 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:42.370125 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:42.370130 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:42.370281 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 02:20:42.370419 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 02:20:42.370676 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 02:20:42.371084 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.358228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49576 (local address 127.4.140.253:38657)
0928 02:20:42.358411 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:42.358417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:42.358456 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:20:42.358519 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:42.358524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:42.358599 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:42.358697 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:42.359484 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.360415 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.361701 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.362267 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.362498 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:42.365498 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:42.365503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:42.365506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:42.365543 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:42.368519 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.369222 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.369227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.369233 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.369369 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.369750 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.369755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.369758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.370007 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:42.370170 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:42.370509 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 02:20:42.370708 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:20:42.371698 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.356117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43972 (local address 127.4.140.254:42559)
0928 02:20:42.356983 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:42.356991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:42.357009 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:20:42.357053 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:42.357057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:42.357105 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:42.357176 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:42.358705 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.359747 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.361233 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.361829 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.361917 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:42.366178 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:42.366182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:42.366185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:42.366213 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:42.368827 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.369553 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.369558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.369561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.369710 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.370091 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.370096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.370100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.370372 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:42.370654 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:42.370941 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 02:20:42.371385 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":544,"threads_started":1}
I20260928 02:20:42.372205 18306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:20:42 dist-test-slave-h37r krb5kdc[18078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562042, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.252@KRBTEST.COM for kudu/127.4.140.252@KRBTEST.COM
Sep 28 02:20:42 dist-test-slave-h37r krb5kdc[18078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.253@KRBTEST.COM
I20260928 02:20:42.386765 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.373089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51852)
0928 02:20:42.373329 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:42.373346 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:42.373437 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:42.373755 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:42.373760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:42.374109 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:42.374233 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:42.374240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.375121 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.375129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:42.376069 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:42.376080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.376831 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.376837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:42.377013 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:42.377553 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:42.377570 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:42.379560 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:42.383494 (+ 3934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:42.383502 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:42.383508 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:42.384120 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:42.384598 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:42.384604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:42.384608 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:42.384771 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:42.385572 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:42.385578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:42.385751 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 02:20:42.385947 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 02:20:42.386265 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260928 02:20:42.387600 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.373072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51852 (local address 127.4.140.252:33581)
0928 02:20:42.373248 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:42.373252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:42.373531 (+ 279us) server_negotiation.cc:407] Connection header received
0928 02:20:42.373570 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:42.373577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:42.373643 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:42.373749 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:42.374358 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.375004 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.376192 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.376671 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.376817 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:42.379729 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:42.379735 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:42.379739 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:42.379773 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:42.383347 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.384245 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.384250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.384254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.384439 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.384880 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.384884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.384888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.385347 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:42.385506 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:42.387045 (+ 1539us) server_negotiation.cc:299] Negotiation successful
0928 02:20:42.387284 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:20:42.389981 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.373490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49580 (local address 127.4.140.253:38657)
0928 02:20:42.373650 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:42.373657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:42.373675 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:20:42.373857 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:42.373862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:42.373928 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:42.374022 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:42.375178 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.375969 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.377248 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.377763 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.377887 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:42.382500 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:42.382508 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:42.382515 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:42.382551 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:42.386516 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.387426 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.387432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.387436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.387846 (+ 410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.388342 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.388348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.388352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.388707 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:42.388911 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:42.389342 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 02:20:42.389609 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:20:42.394207 18306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582: Bootstrap starting.
I20260928 02:20:42.395331 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:20:42.397639 18306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582: Neither blocks nor log segments found. Creating new log.
I20260928 02:20:42.398685 18306 log.cc:824] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582: Log is configured to *not* fsync() on all Append() calls
I20260928 02:20:42.401844 18306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582: No bootstrap required, opened a new log
I20260928 02:20:42.415020 18306 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c67ae843784da8853f9508be04d665" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } } peers { permanent_uuid: "aa007486d16a452e9e12ee39531142d5" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } } peers { permanent_uuid: "ad3d3b3112034b7a89dba60fdc518582" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } }
I20260928 02:20:42.415671 18306 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:20:42.415840 18306 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad3d3b3112034b7a89dba60fdc518582, State: Initialized, Role: FOLLOWER
I20260928 02:20:42.416710 18306 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c67ae843784da8853f9508be04d665" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } } peers { permanent_uuid: "aa007486d16a452e9e12ee39531142d5" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } } peers { permanent_uuid: "ad3d3b3112034b7a89dba60fdc518582" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } }
I20260928 02:20:42.423017 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.402021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.4.140.254:42559)
0928 02:20:42.402214 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:42.402234 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:42.403332 (+ 1098us) server_negotiation.cc:407] Connection header received
0928 02:20:42.403527 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:42.403536 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:42.403632 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:42.403747 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:42.406600 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.407739 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.410238 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.410944 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.411050 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:42.415309 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:42.415319 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:42.415325 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:42.415369 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:42.418789 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.419576 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.419582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.419585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.419749 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.420321 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.420329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.420336 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.420778 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:42.421089 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:42.422134 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0928 02:20:42.422484 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:20:42.423977 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.402488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.4.140.252:33581)
0928 02:20:42.402739 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:42.402747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:42.403888 (+ 1141us) server_negotiation.cc:407] Connection header received
0928 02:20:42.404194 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:42.404199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:42.404291 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:42.404364 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:42.406598 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.407748 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.410250 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.410996 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.411073 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:42.416581 (+ 5508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:42.416591 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:42.416596 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:42.416641 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:42.420325 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.421161 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.421166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.421170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.421336 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.421812 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.421817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.421821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.422223 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:42.422474 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:42.423069 (+ 595us) server_negotiation.cc:299] Negotiation successful
0928 02:20:42.423568 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:20:42.426501 18320 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c67ae843784da8853f9508be04d665" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } } peers { permanent_uuid: "aa007486d16a452e9e12ee39531142d5" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } } peers { permanent_uuid: "ad3d3b3112034b7a89dba60fdc518582" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } } }
I20260928 02:20:42.426916 18320 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:20:42.428253 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.404127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49582 (local address 127.4.140.253:38657)
0928 02:20:42.404296 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:42.404302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:42.411019 (+ 6717us) server_negotiation.cc:407] Connection header received
0928 02:20:42.411400 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:42.411406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:42.411480 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:42.411561 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:42.413865 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.414907 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.416741 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.417430 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.417540 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:42.421058 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:42.421068 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:42.421076 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:42.421128 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:42.424702 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.425805 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.425814 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.425820 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.426015 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.426629 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.426634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.426638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.427043 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:42.427245 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:42.427673 (+ 428us) server_negotiation.cc:299] Negotiation successful
0928 02:20:42.427848 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:20:42.428833 18306 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:20:42.430212 18320 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:20:42.430629 18320 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c67ae843784da8853f9508be04d665" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } } peers { permanent_uuid: "aa007486d16a452e9e12ee39531142d5" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } } peers { permanent_uuid: "ad3d3b3112034b7a89dba60fdc518582" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } }
I20260928 02:20:42.432592 18320 leader_election.cc:288] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1c67ae843784da8853f9508be04d665 (127.4.140.254:42559), aa007486d16a452e9e12ee39531142d5 (127.4.140.253:38657)
W20260928 02:20:42.439466 18331 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:20:42.439608 18331 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:20:42.436224 18239 master_runner.cc:186] Error getting master registration for 127.4.140.254:42559: OK, instance_id { permanent_uuid: "f1c67ae843784da8853f9508be04d665" instance_seqno: 1790562041310292 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:20:42.447522 18239 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:20:42.451706 18158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } attempt: 4
Sep 28 02:20:42 dist-test-slave-h37r krb5kdc[18078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562041, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for kudu/127.4.140.252@KRBTEST.COM
I20260928 02:20:42.468139 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.452288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51876)
0928 02:20:42.452459 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:42.452478 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:42.452582 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:42.452898 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:42.452904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:42.453429 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:42.453629 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:42.453644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.455154 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.455159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:42.456526 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:42.456541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.457547 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.457553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:42.457741 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:42.458470 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:42.458495 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:42.461325 (+ 2830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:42.465218 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:42.465237 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:42.465241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:42.465761 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:42.466211 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:42.466216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:42.466219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:42.466401 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:42.467033 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:42.467038 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:42.467216 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 02:20:42.467408 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 02:20:42.467694 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 02:20:42.468906 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.452348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.4.140.252:33581)
0928 02:20:42.452527 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:42.452533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:42.452553 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:20:42.452684 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:42.452691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:42.452764 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:42.452869 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:42.453766 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.455033 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.456699 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.457405 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.457525 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:42.461503 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:42.461509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:42.461514 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:42.461557 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:42.465056 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.465910 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.465915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.465918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.466076 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.466534 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.466542 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.466548 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.466915 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:42.467055 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:42.468074 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0928 02:20:42.468527 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:20:42.474750 18158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665: Bootstrap starting.
I20260928 02:20:42.478894 18158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665: Neither blocks nor log segments found. Creating new log.
I20260928 02:20:42.480235 18158 log.cc:824] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665: Log is configured to *not* fsync() on all Append() calls
I20260928 02:20:42.483582 18158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665: No bootstrap required, opened a new log
I20260928 02:20:42.494557 18158 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c67ae843784da8853f9508be04d665" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } } peers { permanent_uuid: "aa007486d16a452e9e12ee39531142d5" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } } peers { permanent_uuid: "ad3d3b3112034b7a89dba60fdc518582" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } }
I20260928 02:20:42.494941 18158 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:20:42.495062 18158 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1c67ae843784da8853f9508be04d665, State: Initialized, Role: FOLLOWER
I20260928 02:20:42.495713 18158 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c67ae843784da8853f9508be04d665" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } } peers { permanent_uuid: "aa007486d16a452e9e12ee39531142d5" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } } peers { permanent_uuid: "ad3d3b3112034b7a89dba60fdc518582" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } }
I20260928 02:20:42.496743 18334 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c67ae843784da8853f9508be04d665" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } } peers { permanent_uuid: "aa007486d16a452e9e12ee39531142d5" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } } peers { permanent_uuid: "ad3d3b3112034b7a89dba60fdc518582" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } } }
I20260928 02:20:42.497148 18334 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:20:42.498150 18158 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:20:42.498193 18133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad3d3b3112034b7a89dba60fdc518582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1c67ae843784da8853f9508be04d665" is_pre_election: true
I20260928 02:20:42.498847 18133 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad3d3b3112034b7a89dba60fdc518582 in term 0.
I20260928 02:20:42.499536 18258 leader_election.cc:302] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad3d3b3112034b7a89dba60fdc518582, f1c67ae843784da8853f9508be04d665; no voters:
I20260928 02:20:42.500087 18320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:20:42.500263 18320 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:20:42.500368 18320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:20:42.505012 18320 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c67ae843784da8853f9508be04d665" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } } peers { permanent_uuid: "aa007486d16a452e9e12ee39531142d5" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } } peers { permanent_uuid: "ad3d3b3112034b7a89dba60fdc518582" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } }
I20260928 02:20:42.506537 18133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad3d3b3112034b7a89dba60fdc518582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1c67ae843784da8853f9508be04d665"
I20260928 02:20:42.506752 18133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:20:42.510174 18133 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad3d3b3112034b7a89dba60fdc518582 in term 1.
I20260928 02:20:42.510661 18258 leader_election.cc:302] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad3d3b3112034b7a89dba60fdc518582, f1c67ae843784da8853f9508be04d665; no voters:
I20260928 02:20:42.511245 18320 leader_election.cc:288] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [CANDIDATE]: Term 1 election: Requested vote from peers f1c67ae843784da8853f9508be04d665 (127.4.140.254:42559), aa007486d16a452e9e12ee39531142d5 (127.4.140.253:38657)
I20260928 02:20:42.511941 18320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:20:42.512109 18320 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [term 1 LEADER]: Becoming Leader. State: Replica: ad3d3b3112034b7a89dba60fdc518582, State: Running, Role: LEADER
I20260928 02:20:42.512687 18320 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c67ae843784da8853f9508be04d665" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } } peers { permanent_uuid: "aa007486d16a452e9e12ee39531142d5" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } } peers { permanent_uuid: "ad3d3b3112034b7a89dba60fdc518582" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } }
I20260928 02:20:42.515981 18337 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad3d3b3112034b7a89dba60fdc518582. Latest consensus state: current_term: 1 leader_uuid: "ad3d3b3112034b7a89dba60fdc518582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c67ae843784da8853f9508be04d665" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } } peers { permanent_uuid: "aa007486d16a452e9e12ee39531142d5" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } } peers { permanent_uuid: "ad3d3b3112034b7a89dba60fdc518582" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } } }
I20260928 02:20:42.516477 18337 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [sys.catalog]: This master's current role is: LEADER
I20260928 02:20:42.520335 18339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:20:42.523304 18339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:20:42.540234 18232 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } attempt: 5
I20260928 02:20:42.543717 18133 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665 [term 1 FOLLOWER]: Refusing update from remote peer ad3d3b3112034b7a89dba60fdc518582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:20:42.544909 18337 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1c67ae843784da8853f9508be04d665" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:20:42.550024 18112 tablet.cc:2369] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:20:42 dist-test-slave-h37r krb5kdc[18078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562041, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.253@KRBTEST.COM for kudu/127.4.140.252@KRBTEST.COM
W20260928 02:20:42.556757 18260 tablet.cc:2369] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:20:42.573261 18334 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad3d3b3112034b7a89dba60fdc518582. Latest consensus state: current_term: 1 leader_uuid: "ad3d3b3112034b7a89dba60fdc518582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c67ae843784da8853f9508be04d665" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } } peers { permanent_uuid: "aa007486d16a452e9e12ee39531142d5" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } } peers { permanent_uuid: "ad3d3b3112034b7a89dba60fdc518582" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } } }
I20260928 02:20:42.573585 18334 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:20:42.575026 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.540817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:33581 (local address 127.0.0.1:51884)
0928 02:20:42.541565 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:42.541586 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:42.541695 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:42.542634 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:42.542642 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:42.543331 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:42.543696 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:42.543711 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.545868 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.545876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:42.547840 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:42.547861 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.549432 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.549441 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:42.551434 (+ 1993us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:42.552615 (+ 1181us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:42.552643 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:42.556597 (+ 3954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:42.564870 (+ 8273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:42.564879 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:42.564886 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:42.566195 (+ 1309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:42.567356 (+ 1161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:42.567398 (+ 42us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:42.567405 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:42.567862 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:42.572575 (+ 4713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:42.572581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:42.572938 (+ 357us) client_negotiation.cc:769] Sending connection context
0928 02:20:42.573250 (+ 312us) client_negotiation.cc:240] Negotiation successful
0928 02:20:42.573847 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621}
I20260928 02:20:42.576896 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:42.541889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.4.140.252:33581)
0928 02:20:42.542294 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:42.542300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:42.542318 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:20:42.542369 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:42.542374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:42.542447 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:42.542554 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:42.543959 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.545458 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.548104 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:42.548972 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:42.549131 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:42.556807 (+ 7676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:42.556814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:42.556819 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:42.556860 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:42.564500 (+ 7640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.566577 (+ 2077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.566586 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.566592 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.567090 (+ 498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:42.568351 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:42.568359 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:42.568365 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:42.572341 (+ 3976us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:42.575584 (+ 3243us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:42.575954 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 02:20:42.576442 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293}
I20260928 02:20:42.582458 18232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5: Bootstrap starting.
I20260928 02:20:42.583046 18337 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad3d3b3112034b7a89dba60fdc518582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c67ae843784da8853f9508be04d665" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } } peers { permanent_uuid: "aa007486d16a452e9e12ee39531142d5" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } } peers { permanent_uuid: "ad3d3b3112034b7a89dba60fdc518582" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } } }
I20260928 02:20:42.583309 18337 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [sys.catalog]: This master's current role is: LEADER
I20260928 02:20:42.583568 18347 mvcc.cc:205] Tried to move back new op lower bound from 7334142126216265728 to 7334142126137602048. Current Snapshot: MvccSnapshot[applied={T|T < 7334142126216265728}]
I20260928 02:20:42.584007 18341 mvcc.cc:205] Tried to move back new op lower bound from 7334142126216265728 to 7334142126137602048. Current Snapshot: MvccSnapshot[applied={T|T < 7334142126216265728}]
I20260928 02:20:42.588753 18232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5: Neither blocks nor log segments found. Creating new log.
I20260928 02:20:42.589915 18339 catalog_manager.cc:1347] Generated new cluster ID: 906f310c7d754a898ae45432a19c3933
I20260928 02:20:42.590004 18339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:20:42.591287 18232 log.cc:824] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:20:42.579022 18091 master_runner.cc:186] Error getting master registration for 127.4.140.253:38657: OK, instance_id { permanent_uuid: "aa007486d16a452e9e12ee39531142d5" instance_seqno: 1790562041810453 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:20:42.594352 18091 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:20:42.597657 18334 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad3d3b3112034b7a89dba60fdc518582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c67ae843784da8853f9508be04d665" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } } peers { permanent_uuid: "aa007486d16a452e9e12ee39531142d5" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } } peers { permanent_uuid: "ad3d3b3112034b7a89dba60fdc518582" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } } }
I20260928 02:20:42.597886 18334 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:20:42.603276 18232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5: No bootstrap required, opened a new log
W20260928 02:20:42.604887 18351 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:20:42.605011 18351 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:20:42.607256 18339 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:20:42.609061 18339 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:20:42.614705 18232 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c67ae843784da8853f9508be04d665" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } } peers { permanent_uuid: "aa007486d16a452e9e12ee39531142d5" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } } peers { permanent_uuid: "ad3d3b3112034b7a89dba60fdc518582" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } }
I20260928 02:20:42.615160 18232 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:20:42.615322 18232 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa007486d16a452e9e12ee39531142d5, State: Initialized, Role: FOLLOWER
I20260928 02:20:42.616037 18232 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c67ae843784da8853f9508be04d665" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } } peers { permanent_uuid: "aa007486d16a452e9e12ee39531142d5" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } } peers { permanent_uuid: "ad3d3b3112034b7a89dba60fdc518582" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } }
I20260928 02:20:42.623013 18356 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c67ae843784da8853f9508be04d665" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } } peers { permanent_uuid: "aa007486d16a452e9e12ee39531142d5" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } } peers { permanent_uuid: "ad3d3b3112034b7a89dba60fdc518582" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } } }
I20260928 02:20:42.623478 18356 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:20:42.625470 18232 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:20:42.625499 18207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad3d3b3112034b7a89dba60fdc518582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa007486d16a452e9e12ee39531142d5" is_pre_election: true
I20260928 02:20:42.626070 18207 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad3d3b3112034b7a89dba60fdc518582 in term 0.
I20260928 02:20:42.627312 18205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:20:42.628499 18206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad3d3b3112034b7a89dba60fdc518582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa007486d16a452e9e12ee39531142d5"
I20260928 02:20:42.632351 18205 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5 [term 1 FOLLOWER]: Refusing update from remote peer ad3d3b3112034b7a89dba60fdc518582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:20:42.633154 18337 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa007486d16a452e9e12ee39531142d5" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:20:42.665614 18339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582: Generated new TSK 0
I20260928 02:20:42.666677 18339 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:20:42.681529 18356 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad3d3b3112034b7a89dba60fdc518582. Latest consensus state: current_term: 1 leader_uuid: "ad3d3b3112034b7a89dba60fdc518582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c67ae843784da8853f9508be04d665" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } } peers { permanent_uuid: "aa007486d16a452e9e12ee39531142d5" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } } peers { permanent_uuid: "ad3d3b3112034b7a89dba60fdc518582" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } } }
I20260928 02:20:42.681504 18364 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad3d3b3112034b7a89dba60fdc518582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c67ae843784da8853f9508be04d665" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } } peers { permanent_uuid: "aa007486d16a452e9e12ee39531142d5" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } } peers { permanent_uuid: "ad3d3b3112034b7a89dba60fdc518582" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } } }
I20260928 02:20:42.681919 18356 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:20:42.681945 18364 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:20:42.683347 18320 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad3d3b3112034b7a89dba60fdc518582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c67ae843784da8853f9508be04d665" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42559 } } peers { permanent_uuid: "aa007486d16a452e9e12ee39531142d5" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38657 } } peers { permanent_uuid: "ad3d3b3112034b7a89dba60fdc518582" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 33581 } } }
I20260928 02:20:42.683661 18320 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad3d3b3112034b7a89dba60fdc518582 [sys.catalog]: This master's current role is: LEADER
I20260928 02:20:42.736989 18375 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:20:43.606950 18351 catalog_manager.cc:1259] Loaded cluster ID: 906f310c7d754a898ae45432a19c3933
I20260928 02:20:43.607059 18351 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665: loading cluster ID for follower catalog manager: success
I20260928 02:20:43.609596 18351 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665: acquiring CA information for follower catalog manager: success
I20260928 02:20:43.611332 18351 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f1c67ae843784da8853f9508be04d665: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:20:43.741843 18375 catalog_manager.cc:1259] Loaded cluster ID: 906f310c7d754a898ae45432a19c3933
I20260928 02:20:43.741941 18375 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5: loading cluster ID for follower catalog manager: success
I20260928 02:20:43.744606 18375 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5: acquiring CA information for follower catalog manager: success
I20260928 02:20:43.746441 18375 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aa007486d16a452e9e12ee39531142d5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:20:43.818670 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 18091
I20260928 02:20:43.820029 18153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:20:43.941299 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 18091
I20260928 02:20:43.962337 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 18165
I20260928 02:20:43.963606 18227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:20:44.075564 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 18165
I20260928 02:20:44.095124 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 18239
I20260928 02:20:44.096546 18301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:20:44.204921 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 18239
2026-09-28T02:20:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4677 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:20:44 dist-test-slave-h37r krb5kdc[18383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:20:44 dist-test-slave-h37r krb5kdc[18383](info): set up 2 sockets
Sep 28 02:20:44 dist-test-slave-h37r krb5kdc[18383](info): commencing operation
krb5kdc: starting...
W20260928 02:20:45.316394 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:20:45 dist-test-slave-h37r krb5kdc[18383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:20:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:20:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:20:45.545733 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:40325
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:36631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:40325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:20:45.874086 18396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:45.874481 18396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:45.874658 18396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:45.882103 18396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:20:45.882285 18396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:45.882402 18396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:45.882512 18396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:20:45.882601 18396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:20:45.892758 18396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:40325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:40325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:45.894227 18396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:45.895650 18396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:45.908311 18402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:45.908344 18401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:45.908856 18404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:45.909118 18396 server_base.cc:1034] running on GCE node
I20260928 02:20:45.909940 18396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:45.911394 18396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:45.912635 18396 hybrid_clock.cc:648] HybridClock initialized: now 1790562045912593 us; error 58 us; skew 500 ppm
I20260928 02:20:45.913028 18396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:45 dist-test-slave-h37r krb5kdc[18383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562045, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:45.916201 18396 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:20:45.917801 18396 webserver.cc:460] Webserver started at http://127.4.140.254:37857/ using document root <none> and password file <none>
I20260928 02:20:45.918519 18396 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:45.918670 18396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:20:45.918998 18396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:20:45.921991 18396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "a686cc9e18a340228b0f5343596c78f3"
format_stamp: "Formatted at 2026-09-28 02:20:45 on dist-test-slave-h37r"
server_key: "32b2bd1176e32a582bd68744aefa0ca9"
server_key_iv: "e43bf59670aa3d9bd2299cc56064c811"
server_key_version: "encryptionkey@0"
I20260928 02:20:45.922739 18396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "a686cc9e18a340228b0f5343596c78f3"
format_stamp: "Formatted at 2026-09-28 02:20:45 on dist-test-slave-h37r"
server_key: "32b2bd1176e32a582bd68744aefa0ca9"
server_key_iv: "e43bf59670aa3d9bd2299cc56064c811"
server_key_version: "encryptionkey@0"
I20260928 02:20:45.928231 18396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:20:45.931806 18411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:45.933282 18396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:20:45.933473 18396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "a686cc9e18a340228b0f5343596c78f3"
format_stamp: "Formatted at 2026-09-28 02:20:45 on dist-test-slave-h37r"
server_key: "32b2bd1176e32a582bd68744aefa0ca9"
server_key_iv: "e43bf59670aa3d9bd2299cc56064c811"
server_key_version: "encryptionkey@0"
I20260928 02:20:45.933676 18396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:45.957916 18396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:45.961969 18396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:45.962268 18396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:45.981182 18396 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:40325
I20260928 02:20:45.981194 18462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:40325 every 8 connection(s)
I20260928 02:20:45.982555 18396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:20:45.987349 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:20:45.987977 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 18396
I20260928 02:20:45.988209 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:20:45.988871 4659 external_mini_cluster.cc:1428] Setting key 1898973b5cc9007201fcad6e84d02683
I20260928 02:20:46.001363 18463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3: Bootstrap starting.
Sep 28 02:20:46 dist-test-slave-h37r krb5kdc[18383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:20:46.005470 18463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3: Neither blocks nor log segments found. Creating new log.
I20260928 02:20:46.006526 18463 log.cc:824] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:20:46.009970 18463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3: No bootstrap required, opened a new log
I20260928 02:20:46.013383 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:45.990788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.4.140.254:40325)
0928 02:20:45.991659 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:45.991671 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:45.991716 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:20:45.993453 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:45.993481 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:45.993901 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:45.994235 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:45.996098 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:45.997358 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:45.999328 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:46.000095 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:46.000244 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:46.005427 (+ 5183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:46.005436 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:46.005450 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:46.005490 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:46.009013 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:46.009930 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:46.009937 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:46.009943 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:46.010181 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:46.010623 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:46.010626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:46.010628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:46.011156 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:46.011391 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:46.011859 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 02:20:46.012347 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":378,"threads_started":1}
I20260928 02:20:46.017325 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:20:46.018747 18463 raft_consensus.cc:348] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a686cc9e18a340228b0f5343596c78f3" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 40325 } }
I20260928 02:20:46.019088 18463 raft_consensus.cc:374] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:20:46.019191 18463 raft_consensus.cc:729] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a686cc9e18a340228b0f5343596c78f3, State: Initialized, Role: FOLLOWER
I20260928 02:20:46.019826 18463 consensus_queue.cc:260] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a686cc9e18a340228b0f5343596c78f3" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 40325 } }
I20260928 02:20:46.020162 18463 raft_consensus.cc:388] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:20:46.020339 18463 raft_consensus.cc:482] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:20:46.020519 18463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:20:46.022648 18463 raft_consensus.cc:504] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a686cc9e18a340228b0f5343596c78f3" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 40325 } }
I20260928 02:20:46.023134 18463 leader_election.cc:302] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a686cc9e18a340228b0f5343596c78f3; no voters:
I20260928 02:20:46.023707 18463 leader_election.cc:288] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:20:46.023866 18468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:20:46.024607 18468 raft_consensus.cc:686] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3 [term 1 LEADER]: Becoming Leader. State: Replica: a686cc9e18a340228b0f5343596c78f3, State: Running, Role: LEADER
I20260928 02:20:46.025144 18468 consensus_queue.cc:237] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a686cc9e18a340228b0f5343596c78f3" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 40325 } }
I20260928 02:20:46.025748 18463 sys_catalog.cc:564] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:20:46.028015 18470 sys_catalog.cc:455] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a686cc9e18a340228b0f5343596c78f3. Latest consensus state: current_term: 1 leader_uuid: "a686cc9e18a340228b0f5343596c78f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a686cc9e18a340228b0f5343596c78f3" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 40325 } } }
I20260928 02:20:46.028354 18470 sys_catalog.cc:458] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:20:46.029631 18469 sys_catalog.cc:455] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a686cc9e18a340228b0f5343596c78f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a686cc9e18a340228b0f5343596c78f3" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 40325 } } }
I20260928 02:20:46.030079 18476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:20:46.030742 18469 sys_catalog.cc:458] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:20:46.035048 18476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:20:46.044363 18476 catalog_manager.cc:1347] Generated new cluster ID: 692f9a8c87ac42729c0337d4bd994944
I20260928 02:20:46.044492 18476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:20:46.061120 18476 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:20:46.062508 18476 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:20:46.070309 18476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a686cc9e18a340228b0f5343596c78f3: Generated new TSK 0
I20260928 02:20:46.071329 18476 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:20:46.129485 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 18396
I20260928 02:20:46.130916 18458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:20:46.228503 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 18396
2026-09-28T02:20:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2019 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:20:46 dist-test-slave-h37r krb5kdc[18491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:20:46 dist-test-slave-h37r krb5kdc[18491](info): set up 2 sockets
Sep 28 02:20:46 dist-test-slave-h37r krb5kdc[18491](info): commencing operation
krb5kdc: starting...
W20260928 02:20:47.380460 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:20:47 dist-test-slave-h37r krb5kdc[18491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:20:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:20:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:20:47.621420 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:41423
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:41423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:20:47.951812 18504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:47.952239 18504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:47.952414 18504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:47.959813 18504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:20:47.960022 18504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:47.960105 18504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:47.960184 18504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:20:47.960289 18504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:20:47.970635 18504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:41423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:41423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:47.972162 18504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:47.973512 18504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:47.985672 18509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:47.985728 18510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:47.985756 18512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:47.986485 18504 server_base.cc:1034] running on GCE node
I20260928 02:20:47.987344 18504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:47.988936 18504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:47.990207 18504 hybrid_clock.cc:648] HybridClock initialized: now 1790562047990153 us; error 45 us; skew 500 ppm
I20260928 02:20:47.990761 18504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:47 dist-test-slave-h37r krb5kdc[18491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562047, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:47.995015 18504 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:20:47.996737 18504 webserver.cc:460] Webserver started at http://127.4.140.254:46673/ using document root <none> and password file <none>
I20260928 02:20:47.997473 18504 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:47.997630 18504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:20:47.997949 18504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:20:48.001086 18504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "840ca23fcce445ac9d351c4f3a93e39c"
format_stamp: "Formatted at 2026-09-28 02:20:47 on dist-test-slave-h37r"
server_key: "f0552abebdc67fc16cd1228b0f54ec56"
server_key_iv: "ea528ebd4cb2b5cc07147056868fa098"
server_key_version: "encryptionkey@0"
I20260928 02:20:48.001888 18504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "840ca23fcce445ac9d351c4f3a93e39c"
format_stamp: "Formatted at 2026-09-28 02:20:47 on dist-test-slave-h37r"
server_key: "f0552abebdc67fc16cd1228b0f54ec56"
server_key_iv: "ea528ebd4cb2b5cc07147056868fa098"
server_key_version: "encryptionkey@0"
I20260928 02:20:48.007362 18504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 02:20:48.011040 18519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:48.012544 18504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:20:48.012748 18504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "840ca23fcce445ac9d351c4f3a93e39c"
format_stamp: "Formatted at 2026-09-28 02:20:47 on dist-test-slave-h37r"
server_key: "f0552abebdc67fc16cd1228b0f54ec56"
server_key_iv: "ea528ebd4cb2b5cc07147056868fa098"
server_key_version: "encryptionkey@0"
I20260928 02:20:48.012959 18504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:48.027050 18504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:48.031009 18504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:48.031319 18504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:48.049762 18504 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:41423
I20260928 02:20:48.049793 18570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:41423 every 8 connection(s)
I20260928 02:20:48.051370 18504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:20:48.054108 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 18504
I20260928 02:20:48.054342 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:20:48.054998 4659 external_mini_cluster.cc:1428] Setting key da7f009497ec55eb46fb08a1257ec67c
I20260928 02:20:48.055343 18571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:20:48.069824 18571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c: Bootstrap starting.
Sep 28 02:20:48 dist-test-slave-h37r krb5kdc[18491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:20:48.073153 18571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c: Neither blocks nor log segments found. Creating new log.
I20260928 02:20:48.074366 18571 log.cc:824] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:20:48.077340 18571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c: No bootstrap required, opened a new log
I20260928 02:20:48.079228 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:48.057022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47032 (local address 127.4.140.254:41423)
0928 02:20:48.057817 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:48.057828 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:48.057870 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:20:48.060012 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:48.060035 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:48.060553 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:48.060986 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:48.062753 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:48.064094 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:48.065980 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:48.066737 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:48.066867 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:48.072040 (+ 5173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:48.072049 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:48.072063 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:48.072112 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:48.075198 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:48.075964 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:48.075971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:48.075977 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:48.076153 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:48.076540 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:48.076545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:48.076548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:48.077053 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:48.077399 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:48.077806 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 02:20:48.078251 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":383,"threads_started":1}
I20260928 02:20:48.085769 18571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ca23fcce445ac9d351c4f3a93e39c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 41423 } }
I20260928 02:20:48.086153 18571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:20:48.086266 18571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 840ca23fcce445ac9d351c4f3a93e39c, State: Initialized, Role: FOLLOWER
I20260928 02:20:48.086863 18571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ca23fcce445ac9d351c4f3a93e39c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 41423 } }
I20260928 02:20:48.087111 18571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:20:48.087252 18571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:20:48.087428 18571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:20:48.090224 18571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ca23fcce445ac9d351c4f3a93e39c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 41423 } }
I20260928 02:20:48.090729 18571 leader_election.cc:302] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 840ca23fcce445ac9d351c4f3a93e39c; no voters:
I20260928 02:20:48.091310 18571 leader_election.cc:288] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:20:48.091461 18576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:20:48.092221 18576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 1 LEADER]: Becoming Leader. State: Replica: 840ca23fcce445ac9d351c4f3a93e39c, State: Running, Role: LEADER
I20260928 02:20:48.092758 18576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ca23fcce445ac9d351c4f3a93e39c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 41423 } }
I20260928 02:20:48.093408 18571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:20:48.095695 18578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 840ca23fcce445ac9d351c4f3a93e39c. Latest consensus state: current_term: 1 leader_uuid: "840ca23fcce445ac9d351c4f3a93e39c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ca23fcce445ac9d351c4f3a93e39c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 41423 } } }
I20260928 02:20:48.096082 18578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [sys.catalog]: This master's current role is: LEADER
I20260928 02:20:48.097847 18583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:20:48.097788 18577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "840ca23fcce445ac9d351c4f3a93e39c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ca23fcce445ac9d351c4f3a93e39c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 41423 } } }
I20260928 02:20:48.098206 18577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [sys.catalog]: This master's current role is: LEADER
I20260928 02:20:48.102769 18583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:20:48.110802 18583 catalog_manager.cc:1347] Generated new cluster ID: b705a640770c455285a04c1de807667b
I20260928 02:20:48.110908 18583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:20:48.118095 18583 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:20:48.119942 18583 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:20:48.128798 18583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c: Generated new TSK 0
I20260928 02:20:48.129786 18583 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
I20260928 02:20:48.188717 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:41423
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:20:48.522746 18597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:48.523022 18597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:48.523203 18597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:48.530563 18597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:48.530674 18597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:48.530890 18597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:20:48.540577 18597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:41423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:48.541857 18597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:48.543270 18597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:48.556701 18603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:48.556731 18602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:48.556916 18605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:48.557816 18597 server_base.cc:1034] running on GCE node
I20260928 02:20:48.558449 18597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:48.559448 18597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:48.560674 18597 hybrid_clock.cc:648] HybridClock initialized: now 1790562048560621 us; error 43 us; skew 500 ppm
I20260928 02:20:48.561000 18597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:48 dist-test-slave-h37r krb5kdc[18491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562048, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:48.564337 18597 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:20:48.565932 18597 webserver.cc:460] Webserver started at http://127.4.140.193:33891/ using document root <none> and password file <none>
I20260928 02:20:48.566664 18597 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:48.566774 18597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:20:48.567029 18597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:20:48.569973 18597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "199fb8589bdb46b881841d23d59c85ce"
format_stamp: "Formatted at 2026-09-28 02:20:48 on dist-test-slave-h37r"
server_key: "fcfdef9461a9bfebe90d259964ee0eb1"
server_key_iv: "439bb4699a94c721721230bdeed68aa4"
server_key_version: "encryptionkey@0"
I20260928 02:20:48.570681 18597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "199fb8589bdb46b881841d23d59c85ce"
format_stamp: "Formatted at 2026-09-28 02:20:48 on dist-test-slave-h37r"
server_key: "fcfdef9461a9bfebe90d259964ee0eb1"
server_key_iv: "439bb4699a94c721721230bdeed68aa4"
server_key_version: "encryptionkey@0"
I20260928 02:20:48.575719 18597 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:20:48.579257 18612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:48.580570 18597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:20:48.580880 18597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "199fb8589bdb46b881841d23d59c85ce"
format_stamp: "Formatted at 2026-09-28 02:20:48 on dist-test-slave-h37r"
server_key: "fcfdef9461a9bfebe90d259964ee0eb1"
server_key_iv: "439bb4699a94c721721230bdeed68aa4"
server_key_version: "encryptionkey@0"
I20260928 02:20:48.581079 18597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:48.610621 18597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:48.614655 18597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:48.614948 18597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:48.615999 18597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:20:48.617484 18597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:20:48.617556 18597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:48.617739 18597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:20:48.617816 18597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:48.650257 18597 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:44479
I20260928 02:20:48.650298 18724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:44479 every 8 connection(s)
I20260928 02:20:48.651607 18597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:20:48.652307 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 18597
I20260928 02:20:48.652537 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:20:48.653389 4659 external_mini_cluster.cc:1428] Setting key d6d7c5be4b8395c1c3270fb34ec4249b
Sep 28 02:20:48 dist-test-slave-h37r krb5kdc[18491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562048, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:20:48.676752 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:48.654862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:46433 (local address 127.4.140.254:41423)
0928 02:20:48.655375 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:48.655381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:48.656758 (+ 1377us) server_negotiation.cc:407] Connection header received
0928 02:20:48.659714 (+ 2956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:48.659718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:48.659787 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:48.659884 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:48.662183 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:48.662937 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:48.665068 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:48.665910 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:48.666101 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:48.669800 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:48.669809 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:48.669816 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:48.669860 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:48.673348 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:48.674064 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:48.674073 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:48.674079 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:48.674230 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:48.674640 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:48.674647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:48.674653 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:48.675063 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:48.675217 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:48.675997 (+ 780us) server_negotiation.cc:299] Negotiation successful
0928 02:20:48.676267 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":246,"threads_started":1}
I20260928 02:20:48.677415 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:48.655598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.193:46433)
0928 02:20:48.656563 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:48.656627 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:48.659354 (+ 2727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:48.660187 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:48.660202 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:48.661026 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:48.661896 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:48.661925 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:48.663101 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:48.663109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:48.664916 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:48.664928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:48.666191 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:48.666196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:48.666400 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:48.667105 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:48.667127 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:48.669550 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:48.673479 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:48.673486 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:48.673500 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:48.673918 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:48.674336 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:48.674339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:48.674341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:48.674515 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:48.675183 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:48.675189 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:48.675512 (+ 323us) client_negotiation.cc:769] Sending connection context
0928 02:20:48.675810 (+ 298us) client_negotiation.cc:240] Negotiation successful
0928 02:20:48.676280 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":372,"threads_started":1}
I20260928 02:20:48.679059 18725 heartbeater.cc:347] Connected to a master server at 127.4.140.254:41423
I20260928 02:20:48.679375 18725 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.140.194@KRBTEST.COM; defaulting to no policy
I20260928 02:20:48.680157 18725 heartbeater.cc:511] Master 127.4.140.254:41423 requested a full tablet report, sending...
I20260928 02:20:48.682242 18536 ts_manager.cc:194] Registered new tserver with Master: 199fb8589bdb46b881841d23d59c85ce (127.4.140.193:44479)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.194@KRBTEST.COM" created.
I20260928 02:20:48.684546 18536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:46433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
I20260928 02:20:48.711151 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:41423
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:20:49.061808 18731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:49.062217 18731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:49.062402 18731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:49.071381 18731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:49.071511 18731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:49.071705 18731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:20:49.082201 18731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:41423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:49.083531 18731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:49.085034 18731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:49.099025 18736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:49.098939 18737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:49.098984 18739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:49.100237 18731 server_base.cc:1034] running on GCE node
I20260928 02:20:49.100870 18731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:49.101830 18731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:49.103062 18731 hybrid_clock.cc:648] HybridClock initialized: now 1790562049103003 us; error 51 us; skew 500 ppm
I20260928 02:20:49.103387 18731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:49 dist-test-slave-h37r krb5kdc[18491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562049, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:49.106539 18731 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:20:49.108058 18731 webserver.cc:460] Webserver started at http://127.4.140.194:46133/ using document root <none> and password file <none>
I20260928 02:20:49.108783 18731 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:49.108928 18731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:20:49.109228 18731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:20:49.112301 18731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "e039622a54c24432a124a9c8decca201"
format_stamp: "Formatted at 2026-09-28 02:20:49 on dist-test-slave-h37r"
server_key: "9490986a5277a8a2b9281d368562b6a9"
server_key_iv: "e253371e7022cfd984f1f05c4156b4b6"
server_key_version: "encryptionkey@0"
I20260928 02:20:49.113133 18731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "e039622a54c24432a124a9c8decca201"
format_stamp: "Formatted at 2026-09-28 02:20:49 on dist-test-slave-h37r"
server_key: "9490986a5277a8a2b9281d368562b6a9"
server_key_iv: "e253371e7022cfd984f1f05c4156b4b6"
server_key_version: "encryptionkey@0"
I20260928 02:20:49.118552 18731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:20:49.122076 18746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:49.123416 18731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:20:49.123597 18731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "e039622a54c24432a124a9c8decca201"
format_stamp: "Formatted at 2026-09-28 02:20:49 on dist-test-slave-h37r"
server_key: "9490986a5277a8a2b9281d368562b6a9"
server_key_iv: "e253371e7022cfd984f1f05c4156b4b6"
server_key_version: "encryptionkey@0"
I20260928 02:20:49.123809 18731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:49.145609 18731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:49.149642 18731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:49.149945 18731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:49.150910 18731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:20:49.152360 18731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:20:49.152465 18731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:49.152611 18731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:20:49.152683 18731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:49.185989 18731 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:39979
I20260928 02:20:49.186033 18858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:39979 every 8 connection(s)
I20260928 02:20:49.187256 18731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:20:49.194133 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 18731
I20260928 02:20:49.194330 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:20:49.194900 4659 external_mini_cluster.cc:1428] Setting key bebab240785d82889302371caf489c83
Sep 28 02:20:49 dist-test-slave-h37r krb5kdc[18491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562049, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:20:49.209583 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:49.189921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:44363 (local address 127.4.140.254:41423)
0928 02:20:49.190392 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:49.190398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:49.191372 (+ 974us) server_negotiation.cc:407] Connection header received
0928 02:20:49.193287 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:49.193292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:49.193370 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:49.193466 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:49.195516 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.196467 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.198819 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.199306 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.199404 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:49.203596 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:49.203601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:49.203604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:49.203637 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:49.206433 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:49.207179 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:49.207185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:49.207189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:49.207337 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:49.207795 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:49.207800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:49.207803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:49.208137 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:49.208309 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:49.208983 (+ 674us) server_negotiation.cc:299] Negotiation successful
0928 02:20:49.209209 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":228,"threads_started":1}
I20260928 02:20:49.210569 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:49.190511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.194:44363)
0928 02:20:49.191206 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:49.191247 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:49.192999 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:49.193705 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:49.193732 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:49.194497 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:49.195328 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:49.195356 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.196614 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.196618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:49.198636 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:49.198653 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.199541 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.199550 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:49.199867 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:49.200647 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:49.200689 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:49.203355 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:49.206578 (+ 3223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:49.206587 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:49.206604 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:49.207054 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:49.207474 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:49.207479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:49.207482 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:49.207675 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:49.208275 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:49.208283 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:49.208629 (+ 346us) client_negotiation.cc:769] Sending connection context
0928 02:20:49.208906 (+ 277us) client_negotiation.cc:240] Negotiation successful
0928 02:20:49.209462 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":249,"threads_started":1}
I20260928 02:20:49.212112 18859 heartbeater.cc:347] Connected to a master server at 127.4.140.254:41423
I20260928 02:20:49.212437 18859 heartbeater.cc:464] Registering TS with master...
I20260928 02:20:49.213315 18859 heartbeater.cc:511] Master 127.4.140.254:41423 requested a full tablet report, sending...
I20260928 02:20:49.214910 18536 ts_manager.cc:194] Registered new tserver with Master: e039622a54c24432a124a9c8decca201 (127.4.140.194:39979)
I20260928 02:20:49.215742 18536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:44363
WARNING: no policy specified for kudu/127.4.140.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
I20260928 02:20:49.247408 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:41423
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:20:49.569160 18865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:49.569419 18865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:49.569510 18865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:49.576555 18865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:49.576650 18865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:49.576766 18865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:20:49.586330 18865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:41423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:49.587581 18865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:49.588984 18865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:49.602339 18873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:49.602291 18871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:49.602381 18870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:49.603271 18865 server_base.cc:1034] running on GCE node
I20260928 02:20:49.603847 18865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:49.604851 18865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:49.606071 18865 hybrid_clock.cc:648] HybridClock initialized: now 1790562049606031 us; error 53 us; skew 500 ppm
I20260928 02:20:49.606408 18865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:49 dist-test-slave-h37r krb5kdc[18491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562049, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:49.609519 18865 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:20:49.610903 18865 webserver.cc:460] Webserver started at http://127.4.140.195:35817/ using document root <none> and password file <none>
I20260928 02:20:49.611595 18865 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:49.611743 18865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:20:49.612092 18865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:20:49.614969 18865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c"
format_stamp: "Formatted at 2026-09-28 02:20:49 on dist-test-slave-h37r"
server_key: "16cdc532f6110b1cddc6138989626516"
server_key_iv: "9e0fbf56a5a8734d57d9984faa4026e6"
server_key_version: "encryptionkey@0"
I20260928 02:20:49.615744 18865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c"
format_stamp: "Formatted at 2026-09-28 02:20:49 on dist-test-slave-h37r"
server_key: "16cdc532f6110b1cddc6138989626516"
server_key_iv: "9e0fbf56a5a8734d57d9984faa4026e6"
server_key_version: "encryptionkey@0"
I20260928 02:20:49.620916 18865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:20:49.624226 18880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:49.625566 18865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:20:49.625741 18865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c"
format_stamp: "Formatted at 2026-09-28 02:20:49 on dist-test-slave-h37r"
server_key: "16cdc532f6110b1cddc6138989626516"
server_key_iv: "9e0fbf56a5a8734d57d9984faa4026e6"
server_key_version: "encryptionkey@0"
I20260928 02:20:49.625950 18865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:49.641068 18865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:49.645082 18865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:49.645377 18865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:49.646323 18865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:20:49.647729 18865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:20:49.647825 18865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:49.647987 18865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:20:49.648041 18865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:49.681408 18865 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:33023
I20260928 02:20:49.681448 18992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:33023 every 8 connection(s)
I20260928 02:20:49.682706 18865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:20:49.688479 18725 heartbeater.cc:503] Master 127.4.140.254:41423 was elected leader, sending a full tablet report...
I20260928 02:20:49.688786 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 18865
I20260928 02:20:49.688970 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:20:49.689486 4659 external_mini_cluster.cc:1428] Setting key 3ce7ef18dc3b2136f7ec39a3a3484f3c
Sep 28 02:20:49 dist-test-slave-h37r krb5kdc[18491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562049, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:20:49.703593 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:49.685310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:47649 (local address 127.4.140.254:41423)
0928 02:20:49.685552 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:49.685558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:49.686769 (+ 1211us) server_negotiation.cc:407] Connection header received
0928 02:20:49.688616 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:49.688622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:49.688723 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:49.688838 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:49.690777 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.691876 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.693759 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.694253 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.694386 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:49.698194 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:49.698200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:49.698204 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:49.698237 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:49.700768 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:49.701472 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:49.701476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:49.701478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:49.701612 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:49.701980 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:49.701982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:49.701984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:49.702233 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:49.702366 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:49.703035 (+ 669us) server_negotiation.cc:299] Negotiation successful
0928 02:20:49.703251 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:20:49.704859 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:49.685863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.195:47649)
0928 02:20:49.686602 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:49.686646 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:49.688339 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:49.689090 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:49.689103 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:49.689805 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:49.690575 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:49.690601 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.692066 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.692072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:49.693603 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:49.693615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.694469 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.694476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:49.694708 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:49.695457 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:49.695480 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:49.697946 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:49.700889 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:49.700896 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:49.700910 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:49.701379 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:49.701703 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:49.701706 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:49.701709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:49.701894 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:49.702361 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:49.702367 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:49.702673 (+ 306us) client_negotiation.cc:769] Sending connection context
0928 02:20:49.702923 (+ 250us) client_negotiation.cc:240] Negotiation successful
0928 02:20:49.703430 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":279,"threads_started":1}
I20260928 02:20:49.706204 18993 heartbeater.cc:347] Connected to a master server at 127.4.140.254:41423
I20260928 02:20:49.706501 18993 heartbeater.cc:464] Registering TS with master...
I20260928 02:20:49.707214 18993 heartbeater.cc:511] Master 127.4.140.254:41423 requested a full tablet report, sending...
I20260928 02:20:49.708576 18536 ts_manager.cc:194] Registered new tserver with Master: d77e6699c98b4ca7b4065c17dfaa3e4c (127.4.140.195:33023)
I20260928 02:20:49.709359 18536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:47649
I20260928 02:20:49.717310 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:20:49.733983 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:49.721774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47042 (local address 127.4.140.254:41423)
0928 02:20:49.721969 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:49.721978 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:49.722408 (+ 430us) server_negotiation.cc:407] Connection header received
0928 02:20:49.722541 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:49.722545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:49.722601 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:49.722677 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:49.724050 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.724785 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.726031 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.726457 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.726568 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:49.729012 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:49.729016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:49.729019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:49.729047 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:49.731566 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:49.732255 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:49.732259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:49.732261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:49.732394 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:49.732687 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:49.732690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:49.732692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:49.732999 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:49.733182 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:49.733474 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 02:20:49.733627 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:20:49.737987 18536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:20:49.744990 18536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:20:49.779489 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:49.759118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41368 (local address 127.4.140.193:44479)
0928 02:20:49.770966 (+ 11848us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:49.770977 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:49.771435 (+ 458us) server_negotiation.cc:407] Connection header received
0928 02:20:49.771666 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:49.771673 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:49.771852 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:49.772071 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:20:49.773091 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.774562 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.777270 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.778222 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.778400 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:49.778653 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:49.778949 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 02:20:49.779119 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11664,"thread_start_us":11499,"threads_started":1}
I20260928 02:20:49.779551 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:49.758863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:44479 (local address 127.0.0.1:41368)
0928 02:20:49.771292 (+ 12429us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:49.771311 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:49.771454 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:49.772366 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:49.772371 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:49.772394 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:20:49.772889 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:49.772903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.774710 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.774715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:49.777118 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:49.777128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.778435 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.778440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:49.778641 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:49.778696 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 02:20:49.778882 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 02:20:49.779111 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12292,"mutex_wait_us":25,"thread_start_us":12123,"threads_started":1}
I20260928 02:20:49.780061 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:49.757516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:39979 (local address 127.0.0.1:56972)
0928 02:20:49.769649 (+ 12133us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:49.769675 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:49.769846 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:49.772606 (+ 2760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:49.772612 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:49.772639 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:20:49.773181 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:49.773193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.774671 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.774676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:49.776691 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:49.776709 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.778938 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.778946 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:49.779192 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:49.779307 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 02:20:49.779402 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 02:20:49.779558 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11928,"mutex_wait_us":61,"thread_start_us":11756,"threads_started":1}
I20260928 02:20:49.780540 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:49.757297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:33023 (local address 127.0.0.1:59418)
0928 02:20:49.758309 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:49.758367 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:49.758562 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:49.771288 (+ 12726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:49.771299 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:49.771338 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:20:49.771784 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:49.771797 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.774324 (+ 2527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.774329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:49.776947 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:49.776964 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.778957 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.778963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:49.779313 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:49.779341 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 02:20:49.779418 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:20:49.780171 (+ 753us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":711,"thread_start_us":585,"threads_started":1}
I20260928 02:20:49.780079 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:49.757948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.4.140.195:33023)
0928 02:20:49.770199 (+ 12251us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:49.770212 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:49.770255 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:20:49.770392 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:49.770403 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:49.770805 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:49.771053 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:20:49.771985 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.774170 (+ 2185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.777119 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.778489 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.778599 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:49.778792 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:49.779518 (+ 726us) server_negotiation.cc:299] Negotiation successful
0928 02:20:49.779693 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11993,"thread_start_us":12158,"threads_started":1}
I20260928 02:20:49.781399 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:49.757876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.4.140.194:39979)
0928 02:20:49.771429 (+ 13553us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:49.771437 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:49.771465 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:20:49.771577 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:49.771588 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:49.771856 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:49.772362 (+ 506us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:20:49.773393 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.774516 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.776896 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.778385 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.778528 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:49.778751 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:49.780884 (+ 2133us) server_negotiation.cc:299] Negotiation successful
0928 02:20:49.781063 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13361,"thread_start_us":13581,"threads_started":1}
I20260928 02:20:49.784318 18660 tablet_service.cc:1467] Processing CreateTablet for tablet 2d460d9d05cd49adb16905c941bc07a7 (DEFAULT_TABLE table=test-table [id=94742d0c39664d639d5f4dee88272b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:20:49.784579 18794 tablet_service.cc:1467] Processing CreateTablet for tablet 2d460d9d05cd49adb16905c941bc07a7 (DEFAULT_TABLE table=test-table [id=94742d0c39664d639d5f4dee88272b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:20:49.784826 18928 tablet_service.cc:1467] Processing CreateTablet for tablet 2d460d9d05cd49adb16905c941bc07a7 (DEFAULT_TABLE table=test-table [id=94742d0c39664d639d5f4dee88272b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:20:49.785853 18794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d460d9d05cd49adb16905c941bc07a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:20:49.785863 18660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d460d9d05cd49adb16905c941bc07a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:20:49.786091 18928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d460d9d05cd49adb16905c941bc07a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:20:49.797909 19011 tablet_bootstrap.cc:492] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201: Bootstrap starting.
I20260928 02:20:49.801529 19011 tablet_bootstrap.cc:654] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201: Neither blocks nor log segments found. Creating new log.
I20260928 02:20:49.802556 19011 log.cc:824] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201: Log is configured to *not* fsync() on all Append() calls
I20260928 02:20:49.805397 19011 tablet_bootstrap.cc:492] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201: No bootstrap required, opened a new log
I20260928 02:20:49.805720 19011 ts_tablet_manager.cc:1397] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260928 02:20:49.808694 19013 tablet_bootstrap.cc:492] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c: Bootstrap starting.
I20260928 02:20:49.809185 19012 tablet_bootstrap.cc:492] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce: Bootstrap starting.
I20260928 02:20:49.812505 19013 tablet_bootstrap.cc:654] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c: Neither blocks nor log segments found. Creating new log.
I20260928 02:20:49.812641 19012 tablet_bootstrap.cc:654] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce: Neither blocks nor log segments found. Creating new log.
I20260928 02:20:49.813382 19011 raft_consensus.cc:348] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 } } peers { permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 } } peers { permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 } }
I20260928 02:20:49.813761 19011 raft_consensus.cc:374] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:20:49.813865 19013 log.cc:824] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:20:49.813858 19012 log.cc:824] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce: Log is configured to *not* fsync() on all Append() calls
I20260928 02:20:49.813889 19011 raft_consensus.cc:729] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e039622a54c24432a124a9c8decca201, State: Initialized, Role: FOLLOWER
I20260928 02:20:49.814565 19011 consensus_queue.cc:260] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 } } peers { permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 } } peers { permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 } }
I20260928 02:20:49.819283 19013 tablet_bootstrap.cc:492] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c: No bootstrap required, opened a new log
I20260928 02:20:49.819284 19012 tablet_bootstrap.cc:492] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce: No bootstrap required, opened a new log
I20260928 02:20:49.819599 19013 ts_tablet_manager.cc:1397] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260928 02:20:49.819599 19012 ts_tablet_manager.cc:1397] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20260928 02:20:49.821882 18859 heartbeater.cc:503] Master 127.4.140.254:41423 was elected leader, sending a full tablet report...
I20260928 02:20:49.822944 19011 ts_tablet_manager.cc:1428] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201: Time spent starting tablet: real 0.017s user 0.011s sys 0.006s
I20260928 02:20:49.830720 19012 raft_consensus.cc:348] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 } } peers { permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 } } peers { permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 } }
I20260928 02:20:49.830720 19013 raft_consensus.cc:348] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 } } peers { permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 } } peers { permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 } }
I20260928 02:20:49.831203 19013 raft_consensus.cc:374] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:20:49.831203 19012 raft_consensus.cc:374] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:20:49.831316 19013 raft_consensus.cc:729] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d77e6699c98b4ca7b4065c17dfaa3e4c, State: Initialized, Role: FOLLOWER
I20260928 02:20:49.831316 19012 raft_consensus.cc:729] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 199fb8589bdb46b881841d23d59c85ce, State: Initialized, Role: FOLLOWER
I20260928 02:20:49.832024 19013 consensus_queue.cc:260] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 } } peers { permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 } } peers { permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 } }
I20260928 02:20:49.832026 19012 consensus_queue.cc:260] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 } } peers { permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 } } peers { permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 } }
I20260928 02:20:49.833117 18993 heartbeater.cc:503] Master 127.4.140.254:41423 was elected leader, sending a full tablet report...
I20260928 02:20:49.833976 19013 ts_tablet_manager.cc:1428] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260928 02:20:49.834177 19012 ts_tablet_manager.cc:1428] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260928 02:20:49.904253 18726 tablet.cc:2369] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:20:49.934383 18994 tablet.cc:2369] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:20:49.939414 18860 tablet.cc:2369] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:20:49.977272 19017 raft_consensus.cc:482] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:20:49.977562 19017 raft_consensus.cc:504] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 } } peers { permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 } } peers { permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 } }
I20260928 02:20:49.979137 19017 leader_election.cc:288] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d77e6699c98b4ca7b4065c17dfaa3e4c (127.4.140.195:33023), 199fb8589bdb46b881841d23d59c85ce (127.4.140.193:44479)
I20260928 02:20:49.985917 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:49.979546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:43119 (local address 127.4.140.195:33023)
0928 02:20:49.979731 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:49.979735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:49.980256 (+ 521us) server_negotiation.cc:407] Connection header received
0928 02:20:49.980583 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:49.980589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:49.980668 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:49.980742 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:20:49.981410 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.982142 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.983949 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.984940 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.985140 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:49.985218 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:49.985505 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 02:20:49.985638 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:20:49.986140 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:49.979430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:33023 (local address 127.4.140.194:43119)
0928 02:20:49.980135 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:49.980155 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:49.980318 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:49.980905 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:49.980909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:49.980934 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:20:49.981243 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:49.981250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.982288 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.982294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:49.983770 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:49.983782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.985165 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.985171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:49.985346 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:49.985370 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:20:49.985478 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 02:20:49.985694 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":21,"thread_start_us":337,"threads_started":1}
I20260928 02:20:49.986394 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:49.980519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:35947 (local address 127.4.140.193:44479)
0928 02:20:49.980735 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:49.980741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:49.980936 (+ 195us) server_negotiation.cc:407] Connection header received
0928 02:20:49.981151 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:49.981156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:49.981232 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:49.981330 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:20:49.981856 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.982549 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.984523 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.985422 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.985604 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:49.985675 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:49.986006 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 02:20:49.986131 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 02:20:49.986552 19021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:49.980231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:44479 (local address 127.4.140.194:35947)
0928 02:20:49.980834 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:49.980847 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:49.981009 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:49.981345 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:49.981349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:49.981371 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:20:49.981706 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:49.981721 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.982696 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.982702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:49.984334 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:49.984347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:49.985627 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:49.985634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:49.985853 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:49.985877 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:20:49.985973 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:20:49.986094 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":1,"thread_start_us":292,"threads_started":1}
I20260928 02:20:49.987457 18948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d460d9d05cd49adb16905c941bc07a7" candidate_uuid: "e039622a54c24432a124a9c8decca201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" is_pre_election: true
I20260928 02:20:49.987564 18680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d460d9d05cd49adb16905c941bc07a7" candidate_uuid: "e039622a54c24432a124a9c8decca201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "199fb8589bdb46b881841d23d59c85ce" is_pre_election: true
I20260928 02:20:49.988013 18948 raft_consensus.cc:2454] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e039622a54c24432a124a9c8decca201 in term 0.
I20260928 02:20:49.988104 18680 raft_consensus.cc:2454] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e039622a54c24432a124a9c8decca201 in term 0.
I20260928 02:20:49.988747 18748 leader_election.cc:302] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d77e6699c98b4ca7b4065c17dfaa3e4c, e039622a54c24432a124a9c8decca201; no voters:
I20260928 02:20:49.989130 19017 raft_consensus.cc:2781] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:20:49.989289 19017 raft_consensus.cc:482] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:20:49.989408 19017 raft_consensus.cc:3037] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:20:49.991568 19017 raft_consensus.cc:504] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 } } peers { permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 } } peers { permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 } }
I20260928 02:20:49.992252 19017 leader_election.cc:288] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [CANDIDATE]: Term 1 election: Requested vote from peers d77e6699c98b4ca7b4065c17dfaa3e4c (127.4.140.195:33023), 199fb8589bdb46b881841d23d59c85ce (127.4.140.193:44479)
I20260928 02:20:49.992676 18680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d460d9d05cd49adb16905c941bc07a7" candidate_uuid: "e039622a54c24432a124a9c8decca201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "199fb8589bdb46b881841d23d59c85ce"
I20260928 02:20:49.992673 18948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d460d9d05cd49adb16905c941bc07a7" candidate_uuid: "e039622a54c24432a124a9c8decca201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c"
I20260928 02:20:49.992887 18948 raft_consensus.cc:3037] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:20:49.992887 18680 raft_consensus.cc:3037] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:20:49.996122 18948 raft_consensus.cc:2454] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e039622a54c24432a124a9c8decca201 in term 1.
I20260928 02:20:49.996122 18680 raft_consensus.cc:2454] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e039622a54c24432a124a9c8decca201 in term 1.
I20260928 02:20:49.996647 18748 leader_election.cc:302] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d77e6699c98b4ca7b4065c17dfaa3e4c, e039622a54c24432a124a9c8decca201; no voters:
I20260928 02:20:49.996948 19017 raft_consensus.cc:2781] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:20:49.997398 19017 raft_consensus.cc:686] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [term 1 LEADER]: Becoming Leader. State: Replica: e039622a54c24432a124a9c8decca201, State: Running, Role: LEADER
I20260928 02:20:49.997880 19017 consensus_queue.cc:237] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 } } peers { permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 } } peers { permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 } }
I20260928 02:20:50.003168 18535 catalog_manager.cc:5495] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 reported cstate change: term changed from 0 to 1, leader changed from <none> to e039622a54c24432a124a9c8decca201 (127.4.140.194). New cstate: current_term: 1 leader_uuid: "e039622a54c24432a124a9c8decca201" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 } health_report { overall_health: UNKNOWN } } }
I20260928 02:20:50.024904 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:50.019768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56980 (local address 127.4.140.194:39979)
0928 02:20:50.020043 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:50.020048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:50.020066 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:20:50.020151 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:50.020155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:50.020239 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:50.020359 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:20:50.020677 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:50.021440 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:50.022858 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:50.023241 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:50.023335 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:50.023760 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:20:50.024333 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:20:50.024442 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:50.024536 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 02:20:50.024601 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 02:20:50.033432 18680 raft_consensus.cc:1261] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce [term 1 FOLLOWER]: Refusing update from remote peer e039622a54c24432a124a9c8decca201: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:20:50.033432 18948 raft_consensus.cc:1261] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c [term 1 FOLLOWER]: Refusing update from remote peer e039622a54c24432a124a9c8decca201: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:20:50.034397 19022 consensus_queue.cc:1035] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:20:50.034758 19017 consensus_queue.cc:1035] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [LEADER]: Connected to new peer: Peer: permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:20:50.063088 18536 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:20:50.063403 18536 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:20:50.067137 18536 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c: Sending DeleteTablet for 3 replicas of tablet 2d460d9d05cd49adb16905c941bc07a7
I20260928 02:20:50.068408 18928 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d460d9d05cd49adb16905c941bc07a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:20:50 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:59418
I20260928 02:20:50.068496 18660 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d460d9d05cd49adb16905c941bc07a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:20:50 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:41368
I20260928 02:20:50.069129 18794 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d460d9d05cd49adb16905c941bc07a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:20:50 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:56972
I20260928 02:20:50.069384 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 18597
I20260928 02:20:50.070261 19034 tablet_replica.cc:330] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c: stopping tablet replica
I20260928 02:20:50.070768 19034 raft_consensus.cc:2229] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:20:50.071332 19034 raft_consensus.cc:2258] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:20:50.073179 19036 tablet_replica.cc:330] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201: stopping tablet replica
I20260928 02:20:50.073633 19036 raft_consensus.cc:2229] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:20:50.074378 19036 raft_consensus.cc:2258] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:20:50.076165 19034 ts_tablet_manager.cc:1905] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:20:50.078956 19036 ts_tablet_manager.cc:1905] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:20:50.081617 19035 tablet_replica.cc:330] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce: stopping tablet replica
I20260928 02:20:50.082159 19035 raft_consensus.cc:2229] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:20:50.082691 19035 raft_consensus.cc:2258] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:20:50.083205 19034 ts_tablet_manager.cc:1918] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:20:50.083397 19034 log.cc:1196] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/2d460d9d05cd49adb16905c941bc07a7
I20260928 02:20:50.084002 19034 ts_tablet_manager.cc:1939] T 2d460d9d05cd49adb16905c941bc07a7 P d77e6699c98b4ca7b4065c17dfaa3e4c: Deleting consensus metadata
I20260928 02:20:50.085196 19035 ts_tablet_manager.cc:1905] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:20:50.086120 18521 catalog_manager.cc:4841] TS d77e6699c98b4ca7b4065c17dfaa3e4c (127.4.140.195:33023): tablet 2d460d9d05cd49adb16905c941bc07a7 (table test-table [id=94742d0c39664d639d5f4dee88272b00]) successfully deleted
Sep 28 02:20:50 dist-test-slave-h37r krb5kdc[18491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:20:50.091976 19036 ts_tablet_manager.cc:1918] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:20:50.092216 19036 log.cc:1196] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/2d460d9d05cd49adb16905c941bc07a7
I20260928 02:20:50.092447 19035 ts_tablet_manager.cc:1918] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:20:50.092669 19035 log.cc:1196] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/2d460d9d05cd49adb16905c941bc07a7
I20260928 02:20:50.092895 19036 ts_tablet_manager.cc:1939] T 2d460d9d05cd49adb16905c941bc07a7 P e039622a54c24432a124a9c8decca201: Deleting consensus metadata
I20260928 02:20:50.093349 19035 ts_tablet_manager.cc:1939] T 2d460d9d05cd49adb16905c941bc07a7 P 199fb8589bdb46b881841d23d59c85ce: Deleting consensus metadata
I20260928 02:20:50.095391 18521 catalog_manager.cc:4841] TS 199fb8589bdb46b881841d23d59c85ce (127.4.140.193:44479): tablet 2d460d9d05cd49adb16905c941bc07a7 (table test-table [id=94742d0c39664d639d5f4dee88272b00]) successfully deleted
I20260928 02:20:50.095602 18520 catalog_manager.cc:4841] TS e039622a54c24432a124a9c8decca201 (127.4.140.194:39979): tablet 2d460d9d05cd49adb16905c941bc07a7 (table test-table [id=94742d0c39664d639d5f4dee88272b00]) successfully deleted
I20260928 02:20:50.100004 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:50.070668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.4.140.193:44479)
0928 02:20:50.070862 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:50.070868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:50.072494 (+ 1626us) server_negotiation.cc:407] Connection header received
0928 02:20:50.072700 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:50.072707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:50.072798 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:50.072901 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:50.076460 (+ 3559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:50.077639 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:50.083132 (+ 5493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:50.083820 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:50.083963 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:50.092272 (+ 8309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:50.092288 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:50.092308 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:50.092362 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:50.096494 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:50.097217 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:50.097227 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:50.097231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:50.097399 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:50.098049 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:50.098055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:50.098059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:50.098556 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:50.098871 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:50.099326 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 02:20:50.099558 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:20:50.100678 18720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:20:50.191601 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 18597
I20260928 02:20:50.213066 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 18731
Sep 28 02:20:50 dist-test-slave-h37r krb5kdc[18491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.194@KRBTEST.COM
I20260928 02:20:50.232368 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:50.214036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.4.140.194:39979)
0928 02:20:50.214256 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:50.214264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:50.214373 (+ 109us) server_negotiation.cc:407] Connection header received
0928 02:20:50.214634 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:50.214640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:50.214712 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:50.214837 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:50.216353 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:50.217546 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:50.219156 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:50.219823 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:50.220251 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:50.225184 (+ 4933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:50.225200 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:50.225226 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:50.225275 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:50.229368 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:50.230088 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:50.230098 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:50.230102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:50.230274 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:50.230718 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:50.230725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:50.230729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:50.231219 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:50.231550 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:50.231711 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 02:20:50.231963 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 02:20:50.233188 18854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:20:50.346879 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 18731
I20260928 02:20:50.365310 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 18865
Sep 28 02:20:50 dist-test-slave-h37r krb5kdc[18491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.195@KRBTEST.COM
I20260928 02:20:50.379702 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:50.366077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59420 (local address 127.4.140.195:33023)
0928 02:20:50.366276 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:50.366280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:50.366299 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:20:50.366429 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:50.366433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:50.366494 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:50.366574 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:50.367828 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:50.368573 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:50.369855 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:50.370223 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:50.370331 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:50.374555 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:50.374570 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:50.374586 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:50.374623 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:50.377284 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:50.377906 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:50.377914 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:50.377916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:50.378064 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:50.378375 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:50.378378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:50.378380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:50.378704 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:50.378927 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:50.379163 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 02:20:50.379320 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:20:50.380640 18988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:20:50.470592 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 18865
I20260928 02:20:50.487172 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 18504
I20260928 02:20:50.488416 18566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:20:50.598625 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 18504
I20260928 02:20:50.620934 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:41423
--webserver_interface=127.4.140.254
--webserver_port=46673
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:41423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:20:50.947290 19042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:50.947674 19042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:50.947846 19042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:50.955090 19042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:20:50.955199 19042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:50.955302 19042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:50.955410 19042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:20:50.955472 19042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:20:50.965739 19042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:41423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:41423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=46673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:50.967134 19042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:50.968536 19042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:50.991214 19048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:50.991166 19049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:50.991182 19051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:50.992038 19042 server_base.cc:1034] running on GCE node
I20260928 02:20:50.992867 19042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:50.994449 19042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:50.995659 19042 hybrid_clock.cc:648] HybridClock initialized: now 1790562050995591 us; error 63 us; skew 500 ppm
I20260928 02:20:50.996109 19042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:50 dist-test-slave-h37r krb5kdc[18491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562050, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:50.999620 19042 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:20:51.001430 19042 webserver.cc:460] Webserver started at http://127.4.140.254:46673/ using document root <none> and password file <none>
I20260928 02:20:51.002473 19042 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:51.002637 19042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:20:51.009243 19042 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 02:20:51.012189 19058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:51.013566 19042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:20:51.013756 19042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "840ca23fcce445ac9d351c4f3a93e39c"
format_stamp: "Formatted at 2026-09-28 02:20:47 on dist-test-slave-h37r"
server_key: "f0552abebdc67fc16cd1228b0f54ec56"
server_key_iv: "ea528ebd4cb2b5cc07147056868fa098"
server_key_version: "encryptionkey@0"
I20260928 02:20:51.014824 19042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:51.037568 19042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:51.041908 19042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:51.042202 19042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:51.061623 19042 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:41423
I20260928 02:20:51.061656 19109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:41423 every 8 connection(s)
I20260928 02:20:51.063206 19042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:20:51.064754 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 19042
I20260928 02:20:51.065764 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:44479
--local_ip_for_outbound_sockets=127.4.140.193
--tserver_master_addrs=127.4.140.254:41423
--webserver_port=33891
--webserver_interface=127.4.140.193
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:20:51.073601 19110 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:20:51.077128 19110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c: Bootstrap starting.
I20260928 02:20:51.096873 19110 log.cc:824] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:20:51.109465 19110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:20:51.110042 19110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c: Bootstrap complete.
I20260928 02:20:51.120116 19110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ca23fcce445ac9d351c4f3a93e39c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 41423 } }
I20260928 02:20:51.120965 19110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 840ca23fcce445ac9d351c4f3a93e39c, State: Initialized, Role: FOLLOWER
I20260928 02:20:51.121608 19110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ca23fcce445ac9d351c4f3a93e39c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 41423 } }
I20260928 02:20:51.121850 19110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:20:51.121966 19110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:20:51.122169 19110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:20:51.126019 19110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ca23fcce445ac9d351c4f3a93e39c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 41423 } }
I20260928 02:20:51.126565 19110 leader_election.cc:302] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 840ca23fcce445ac9d351c4f3a93e39c; no voters:
I20260928 02:20:51.127214 19110 leader_election.cc:288] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:20:51.127347 19114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:20:51.128651 19114 raft_consensus.cc:686] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 2 LEADER]: Becoming Leader. State: Replica: 840ca23fcce445ac9d351c4f3a93e39c, State: Running, Role: LEADER
I20260928 02:20:51.129189 19114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ca23fcce445ac9d351c4f3a93e39c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 41423 } }
I20260928 02:20:51.129551 19110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:20:51.132834 19115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "840ca23fcce445ac9d351c4f3a93e39c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ca23fcce445ac9d351c4f3a93e39c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 41423 } } }
I20260928 02:20:51.133250 19115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [sys.catalog]: This master's current role is: LEADER
I20260928 02:20:51.133687 19116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 840ca23fcce445ac9d351c4f3a93e39c. Latest consensus state: current_term: 2 leader_uuid: "840ca23fcce445ac9d351c4f3a93e39c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ca23fcce445ac9d351c4f3a93e39c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 41423 } } }
I20260928 02:20:51.133962 19116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [sys.catalog]: This master's current role is: LEADER
I20260928 02:20:51.144434 19123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:20:51.152814 19123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:20:51.154132 19123 catalog_manager.cc:1259] Loaded cluster ID: b705a640770c455285a04c1de807667b
I20260928 02:20:51.154233 19123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:20:51.157173 19123 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 02:20:51.177816 19123 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 02:20:51.202474 19123 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:20:51.204800 19123 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c: Loaded TSK: 0
I20260928 02:20:51.207294 19123 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:20:51.441315 19112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:51.441629 19112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:51.441787 19112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:51.449204 19112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:51.449362 19112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:51.449528 19112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:20:51.460090 19112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:44479
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=33891
--tserver_master_addrs=127.4.140.254:41423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:51.461405 19112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:51.462810 19112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:51.478149 19137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:51.478192 19140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:51.478060 19138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:51.478947 19112 server_base.cc:1034] running on GCE node
I20260928 02:20:51.479573 19112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:51.480681 19112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:51.481913 19112 hybrid_clock.cc:648] HybridClock initialized: now 1790562051481857 us; error 52 us; skew 500 ppm
I20260928 02:20:51.482245 19112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:51 dist-test-slave-h37r krb5kdc[18491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562051, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:51.489878 19112 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:20:51.491498 19112 webserver.cc:460] Webserver started at http://127.4.140.193:33891/ using document root <none> and password file <none>
I20260928 02:20:51.492319 19112 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:51.492465 19112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:20:51.498255 19112 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:20:51.501283 19147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:51.502817 19112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:20:51.503005 19112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "199fb8589bdb46b881841d23d59c85ce"
format_stamp: "Formatted at 2026-09-28 02:20:48 on dist-test-slave-h37r"
server_key: "fcfdef9461a9bfebe90d259964ee0eb1"
server_key_iv: "439bb4699a94c721721230bdeed68aa4"
server_key_version: "encryptionkey@0"
I20260928 02:20:51.504076 19112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:51.523268 19112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:51.527709 19112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:51.528018 19112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:51.529112 19112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:20:51.530704 19112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:20:51.530807 19112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:51.530952 19112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:20:51.531047 19112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:51.566563 19112 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:44479
I20260928 02:20:51.566627 19259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:44479 every 8 connection(s)
I20260928 02:20:51.568135 19112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:20:51.572446 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 19112
I20260928 02:20:51.573076 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:39979
--local_ip_for_outbound_sockets=127.4.140.194
--tserver_master_addrs=127.4.140.254:41423
--webserver_port=46133
--webserver_interface=127.4.140.194
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:20:51 dist-test-slave-h37r krb5kdc[18491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562051, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:20:51.593668 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:51.571011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:57495 (local address 127.4.140.254:41423)
0928 02:20:51.571585 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:51.571596 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:51.572409 (+ 813us) server_negotiation.cc:407] Connection header received
0928 02:20:51.574608 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:51.574628 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:51.575054 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:51.575471 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:51.576917 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:51.578927 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:51.580883 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:51.581507 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:51.581652 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:51.586043 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:51.586055 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:51.586079 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:51.586125 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:51.589046 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:51.589986 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:51.589993 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:51.589998 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:51.590163 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:51.590664 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:51.590667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:51.590669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:51.591227 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:51.591377 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:51.592404 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0928 02:20:51.592739 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":277,"threads_started":1}
I20260928 02:20:51.594347 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:51.571349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.193:57495)
0928 02:20:51.572210 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:51.572274 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:51.574057 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:51.575572 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:51.575579 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:51.576143 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:51.576671 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:51.576689 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:51.579094 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:51.579098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:51.580706 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:51.580736 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:51.581726 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:51.581735 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:51.582044 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:51.582738 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:51.582788 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:51.585726 (+ 2938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:51.589189 (+ 3463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:51.589201 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:51.589226 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:51.589852 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:51.590283 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:51.590291 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:51.590297 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:51.590543 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:51.591377 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:51.591389 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:51.591867 (+ 478us) client_negotiation.cc:769] Sending connection context
0928 02:20:51.592239 (+ 372us) client_negotiation.cc:240] Negotiation successful
0928 02:20:51.592870 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":365,"threads_started":1}
I20260928 02:20:51.597354 19260 heartbeater.cc:347] Connected to a master server at 127.4.140.254:41423
I20260928 02:20:51.597703 19260 heartbeater.cc:464] Registering TS with master...
I20260928 02:20:51.598448 19260 heartbeater.cc:511] Master 127.4.140.254:41423 requested a full tablet report, sending...
I20260928 02:20:51.600785 19075 ts_manager.cc:194] Registered new tserver with Master: 199fb8589bdb46b881841d23d59c85ce (127.4.140.193:44479)
I20260928 02:20:51.602306 19075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:57495
W20260928 02:20:51.900431 19264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:51.900775 19264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:51.900934 19264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:51.908200 19264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:51.908347 19264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:51.908531 19264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:20:51.918232 19264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:39979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=46133
--tserver_master_addrs=127.4.140.254:41423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:51.919760 19264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:51.921119 19264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:51.933920 19269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:51.933890 19270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:51.934340 19272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:51.934798 19264 server_base.cc:1034] running on GCE node
I20260928 02:20:51.935381 19264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:51.936353 19264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:51.937589 19264 hybrid_clock.cc:648] HybridClock initialized: now 1790562051937542 us; error 56 us; skew 500 ppm
I20260928 02:20:51.937901 19264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:51 dist-test-slave-h37r krb5kdc[18491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562051, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:51.947482 19264 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:20:51.948995 19264 webserver.cc:460] Webserver started at http://127.4.140.194:46133/ using document root <none> and password file <none>
I20260928 02:20:51.949819 19264 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:51.949970 19264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:20:51.955358 19264 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260928 02:20:51.958202 19279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:51.959566 19264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:20:51.959792 19264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "e039622a54c24432a124a9c8decca201"
format_stamp: "Formatted at 2026-09-28 02:20:49 on dist-test-slave-h37r"
server_key: "9490986a5277a8a2b9281d368562b6a9"
server_key_iv: "e253371e7022cfd984f1f05c4156b4b6"
server_key_version: "encryptionkey@0"
I20260928 02:20:51.960860 19264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:51.977368 19264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:51.981299 19264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:51.981596 19264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:51.982594 19264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:20:51.983870 19264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:20:51.984035 19264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:51.984163 19264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:20:51.984254 19264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:52.028251 19264 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:39979
I20260928 02:20:52.028329 19391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:39979 every 8 connection(s)
I20260928 02:20:52.029853 19264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:20:52.039536 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 19264
I20260928 02:20:52.040313 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:33023
--local_ip_for_outbound_sockets=127.4.140.195
--tserver_master_addrs=127.4.140.254:41423
--webserver_port=35817
--webserver_interface=127.4.140.195
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:20:52 dist-test-slave-h37r krb5kdc[18491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562051, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:20:52.058557 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:52.034585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:51881 (local address 127.4.140.254:41423)
0928 02:20:52.034800 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:52.034806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:52.035087 (+ 281us) server_negotiation.cc:407] Connection header received
0928 02:20:52.038335 (+ 3248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:52.038341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:52.038429 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:52.038555 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:52.040749 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.042018 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.044600 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.045398 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.045557 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:52.050206 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:52.050216 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:52.050223 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:52.050285 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:52.055090 (+ 4805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:52.055826 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:52.055834 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:52.055840 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:52.056057 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:52.056509 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:52.056516 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:52.056522 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:52.056929 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:52.057084 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:52.057826 (+ 742us) server_negotiation.cc:299] Negotiation successful
0928 02:20:52.058094 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:20:52.059167 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:52.034045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.194:51881)
0928 02:20:52.034914 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:52.034961 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:52.037971 (+ 3010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:52.038826 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:52.038837 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:52.039663 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:52.040523 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:52.040552 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.042186 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.042194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:52.044364 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:52.044382 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.045654 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.045663 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:52.046023 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:52.046981 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:52.047016 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:52.049914 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:52.055255 (+ 5341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:52.055262 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:52.055275 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:52.055683 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:52.056166 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:52.056170 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:52.056172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:52.056376 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:52.057057 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:52.057063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:52.057396 (+ 333us) client_negotiation.cc:769] Sending connection context
0928 02:20:52.057632 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 02:20:52.058031 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":498,"threads_started":1}
I20260928 02:20:52.060818 19392 heartbeater.cc:347] Connected to a master server at 127.4.140.254:41423
I20260928 02:20:52.061164 19392 heartbeater.cc:464] Registering TS with master...
I20260928 02:20:52.062153 19392 heartbeater.cc:511] Master 127.4.140.254:41423 requested a full tablet report, sending...
I20260928 02:20:52.063972 19075 ts_manager.cc:194] Registered new tserver with Master: e039622a54c24432a124a9c8decca201 (127.4.140.194:39979)
I20260928 02:20:52.065160 19075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:51881
W20260928 02:20:52.367712 19395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:52.368076 19395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:52.368266 19395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:52.375553 19395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:52.375691 19395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:52.375871 19395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:20:52.385846 19395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:33023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=35817
--tserver_master_addrs=127.4.140.254:41423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:52.387135 19395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:52.388629 19395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:52.401700 19401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:52.401684 19403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:52.401678 19400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:52.402407 19395 server_base.cc:1034] running on GCE node
I20260928 02:20:52.403414 19395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:52.404774 19395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:52.405968 19395 hybrid_clock.cc:648] HybridClock initialized: now 1790562052405914 us; error 44 us; skew 500 ppm
I20260928 02:20:52.406358 19395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:52 dist-test-slave-h37r krb5kdc[18491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562052, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:52.410573 19395 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:20:52.411952 19395 webserver.cc:460] Webserver started at http://127.4.140.195:35817/ using document root <none> and password file <none>
I20260928 02:20:52.412662 19395 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:52.412796 19395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:20:52.418090 19395 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 02:20:52.420956 19410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:52.422240 19395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:20:52.422402 19395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c"
format_stamp: "Formatted at 2026-09-28 02:20:49 on dist-test-slave-h37r"
server_key: "16cdc532f6110b1cddc6138989626516"
server_key_iv: "9e0fbf56a5a8734d57d9984faa4026e6"
server_key_version: "encryptionkey@0"
I20260928 02:20:52.423314 19395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:52.437661 19395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:52.441483 19395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:52.441743 19395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:52.442683 19395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:20:52.444008 19395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:20:52.444119 19395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:52.444247 19395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:20:52.444339 19395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:52.478238 19395 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:33023
I20260928 02:20:52.478315 19522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:33023 every 8 connection(s)
I20260928 02:20:52.479704 19395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:20:52.486843 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 19395
Sep 28 02:20:52 dist-test-slave-h37r krb5kdc[18491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562052, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:20:52.501760 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:52.482312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:59847 (local address 127.4.140.254:41423)
0928 02:20:52.482532 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:52.482539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:52.483660 (+ 1121us) server_negotiation.cc:407] Connection header received
0928 02:20:52.485496 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:52.485500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:52.485569 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:52.485661 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:52.487149 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.488088 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.490551 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.491095 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.491237 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:52.495032 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:52.495039 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:52.495044 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:52.495085 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:52.498770 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:52.499507 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:52.499514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:52.499520 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:52.499688 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:52.500219 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:52.500224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:52.500227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:52.500585 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:52.500745 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:52.501388 (+ 643us) server_negotiation.cc:299] Negotiation successful
0928 02:20:52.501534 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:20:52.503037 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:52.482819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.195:59847)
0928 02:20:52.483504 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:52.483541 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:52.485243 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:52.485858 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:52.485866 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:52.486372 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:20:52.486955 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:52.486979 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.488320 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.488327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:52.490395 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:52.490411 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.491306 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.491311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:52.491506 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:52.492256 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:20:52.492286 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:20:52.494830 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:20:52.498909 (+ 4079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:52.498922 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:52.498938 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:52.499381 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:52.499838 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:20:52.499843 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:20:52.499847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:20:52.500102 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:20:52.500718 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:20:52.500724 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:20:52.501020 (+ 296us) client_negotiation.cc:769] Sending connection context
0928 02:20:52.501283 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 02:20:52.501820 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":252,"threads_started":1}
I20260928 02:20:52.504443 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:52.487735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.4.140.254:41423)
0928 02:20:52.488283 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:52.488288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:52.488808 (+ 520us) server_negotiation.cc:407] Connection header received
0928 02:20:52.488975 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:52.488978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:52.489041 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:52.489135 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:52.490990 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.491934 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.493968 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.494543 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.494677 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:52.498042 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:52.498050 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:52.498056 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:52.498110 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:52.501125 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:52.502118 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:52.502122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:52.502124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:52.502225 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:52.502647 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:52.502650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:52.502652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:52.502873 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:52.503058 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:52.503789 (+ 731us) server_negotiation.cc:299] Negotiation successful
0928 02:20:52.504030 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":288,"threads_started":1}
I20260928 02:20:52.504737 19523 heartbeater.cc:347] Connected to a master server at 127.4.140.254:41423
I20260928 02:20:52.505132 19523 heartbeater.cc:464] Registering TS with master...
I20260928 02:20:52.505939 19523 heartbeater.cc:511] Master 127.4.140.254:41423 requested a full tablet report, sending...
I20260928 02:20:52.507454 19075 ts_manager.cc:194] Registered new tserver with Master: d77e6699c98b4ca7b4065c17dfaa3e4c (127.4.140.195:33023)
I20260928 02:20:52.508342 19075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:59847
I20260928 02:20:52.519165 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:20:52.525748 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:52.520100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47060 (local address 127.4.140.254:41423)
0928 02:20:52.520288 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:52.520293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:52.520737 (+ 444us) server_negotiation.cc:407] Connection header received
0928 02:20:52.520907 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:52.520910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:52.521056 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:52.521180 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:20:52.521570 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.522262 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.523713 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.524189 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.524288 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:52.524751 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:20:52.525252 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:20:52.525357 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:52.525499 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 02:20:52.525562 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:20:52.526590 19075 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:20:52.533910 19075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:20:52.558424 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:52.546335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56996 (local address 127.4.140.194:39979)
0928 02:20:52.548068 (+ 1733us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:52.548078 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:52.548113 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:20:52.548220 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:52.548226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:52.548498 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:52.549598 (+ 1100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:20:52.549951 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.551599 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.555637 (+ 4038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.557199 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.557355 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:52.557602 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:52.557881 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 02:20:52.558037 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1534,"thread_start_us":1392,"threads_started":1}
I20260928 02:20:52.558477 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:52.545899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:39979 (local address 127.0.0.1:56996)
0928 02:20:52.546811 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:52.546853 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:52.547020 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:52.548743 (+ 1723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:52.548750 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:52.548790 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:20:52.549315 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:52.549333 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.551800 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.551808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:52.555447 (+ 3639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:52.555468 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.557422 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.557429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:52.557624 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:52.557680 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 02:20:52.557831 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 02:20:52.558029 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":28,"thread_start_us":490,"threads_started":1}
I20260928 02:20:52.558701 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:52.546767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41388 (local address 127.4.140.193:44479)
0928 02:20:52.547544 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:52.547554 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:52.548109 (+ 555us) server_negotiation.cc:407] Connection header received
0928 02:20:52.548410 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:52.548419 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:52.548695 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:52.548919 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:20:52.550024 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.552202 (+ 2178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.556025 (+ 3823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.557234 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.557379 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:52.557593 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:52.558177 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 02:20:52.558348 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":546,"threads_started":1}
I20260928 02:20:52.559551 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:52.546168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:44479 (local address 127.0.0.1:41388)
0928 02:20:52.547995 (+ 1827us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:52.548014 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:52.548213 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:52.549260 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:52.549268 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:52.549304 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:20:52.549781 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:52.549796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.552440 (+ 2644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.552448 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:52.555817 (+ 3369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:52.555830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.557762 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.557770 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:52.557926 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:52.557962 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:20:52.559174 (+ 1212us) client_negotiation.cc:240] Negotiation successful
0928 02:20:52.559284 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1694,"mutex_wait_us":251,"thread_start_us":1826,"threads_started":1}
I20260928 02:20:52.562700 19327 tablet_service.cc:1467] Processing CreateTablet for tablet 9da613610d334ae5a0135d2b66d2898f (DEFAULT_TABLE table=test-table [id=98d5790f060a4459937d010d0b59ef2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:20:52.563406 19195 tablet_service.cc:1467] Processing CreateTablet for tablet 9da613610d334ae5a0135d2b66d2898f (DEFAULT_TABLE table=test-table [id=98d5790f060a4459937d010d0b59ef2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:20:52.564301 19327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9da613610d334ae5a0135d2b66d2898f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:20:52.564306 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:52.548345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:33023 (local address 127.0.0.1:59428)
0928 02:20:52.555515 (+ 7170us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:52.555530 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:52.555665 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:52.556885 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:52.556891 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:52.556919 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:20:52.557317 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:52.557332 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.560082 (+ 2750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.560089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:52.561751 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:52.561765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.563517 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.563524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:52.563719 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:52.563751 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:20:52.563824 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:20:52.563987 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7070,"thread_start_us":6833,"threads_started":1}
I20260928 02:20:52.564867 19195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9da613610d334ae5a0135d2b66d2898f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:20:52.565320 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:52.548727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59428 (local address 127.4.140.195:33023)
0928 02:20:52.556265 (+ 7538us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:52.556273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:52.556307 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:20:52.556404 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:52.556412 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:52.556659 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:52.556876 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:20:52.557532 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.559937 (+ 2405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.561922 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.563263 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.564245 (+ 982us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:52.564659 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:52.564738 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 02:20:52.564894 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7362,"thread_start_us":7198,"threads_started":1}
I20260928 02:20:52.568774 19458 tablet_service.cc:1467] Processing CreateTablet for tablet 9da613610d334ae5a0135d2b66d2898f (DEFAULT_TABLE table=test-table [id=98d5790f060a4459937d010d0b59ef2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:20:52.570124 19458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9da613610d334ae5a0135d2b66d2898f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:20:52.580610 19539 tablet_bootstrap.cc:492] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201: Bootstrap starting.
I20260928 02:20:52.581805 19540 tablet_bootstrap.cc:492] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce: Bootstrap starting.
I20260928 02:20:52.584609 19541 tablet_bootstrap.cc:492] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c: Bootstrap starting.
I20260928 02:20:52.584972 19540 tablet_bootstrap.cc:654] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce: Neither blocks nor log segments found. Creating new log.
I20260928 02:20:52.585899 19539 tablet_bootstrap.cc:654] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201: Neither blocks nor log segments found. Creating new log.
I20260928 02:20:52.586318 19540 log.cc:824] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce: Log is configured to *not* fsync() on all Append() calls
I20260928 02:20:52.587013 19539 log.cc:824] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201: Log is configured to *not* fsync() on all Append() calls
I20260928 02:20:52.588428 19541 tablet_bootstrap.cc:654] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c: Neither blocks nor log segments found. Creating new log.
I20260928 02:20:52.589533 19540 tablet_bootstrap.cc:492] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce: No bootstrap required, opened a new log
I20260928 02:20:52.589656 19541 log.cc:824] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:20:52.589850 19540 ts_tablet_manager.cc:1397] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260928 02:20:52.590205 19539 tablet_bootstrap.cc:492] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201: No bootstrap required, opened a new log
I20260928 02:20:52.590473 19539 ts_tablet_manager.cc:1397] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 02:20:52.592806 19541 tablet_bootstrap.cc:492] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c: No bootstrap required, opened a new log
I20260928 02:20:52.593066 19541 ts_tablet_manager.cc:1397] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 02:20:52.597952 19540 raft_consensus.cc:348] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 } } peers { permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 } } peers { permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 } }
I20260928 02:20:52.598362 19540 raft_consensus.cc:374] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:20:52.598459 19540 raft_consensus.cc:729] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 199fb8589bdb46b881841d23d59c85ce, State: Initialized, Role: FOLLOWER
I20260928 02:20:52.599117 19540 consensus_queue.cc:260] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 } } peers { permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 } } peers { permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 } }
I20260928 02:20:52.600279 19260 heartbeater.cc:503] Master 127.4.140.254:41423 was elected leader, sending a full tablet report...
I20260928 02:20:52.600991 19540 ts_tablet_manager.cc:1428] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260928 02:20:52.601094 19539 raft_consensus.cc:348] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 } } peers { permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 } } peers { permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 } }
I20260928 02:20:52.601513 19539 raft_consensus.cc:374] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:20:52.601613 19539 raft_consensus.cc:729] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e039622a54c24432a124a9c8decca201, State: Initialized, Role: FOLLOWER
I20260928 02:20:52.602203 19539 consensus_queue.cc:260] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 } } peers { permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 } } peers { permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 } }
I20260928 02:20:52.603360 19392 heartbeater.cc:503] Master 127.4.140.254:41423 was elected leader, sending a full tablet report...
I20260928 02:20:52.603973 19539 ts_tablet_manager.cc:1428] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260928 02:20:52.604471 19541 raft_consensus.cc:348] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 } } peers { permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 } } peers { permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 } }
I20260928 02:20:52.605036 19541 raft_consensus.cc:374] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:20:52.605162 19541 raft_consensus.cc:729] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d77e6699c98b4ca7b4065c17dfaa3e4c, State: Initialized, Role: FOLLOWER
I20260928 02:20:52.605791 19541 consensus_queue.cc:260] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 } } peers { permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 } } peers { permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 } }
I20260928 02:20:52.606849 19523 heartbeater.cc:503] Master 127.4.140.254:41423 was elected leader, sending a full tablet report...
I20260928 02:20:52.607486 19541 ts_tablet_manager.cc:1428] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260928 02:20:52.731334 19524 tablet.cc:2369] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:20:52.782213 19393 tablet.cc:2369] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:20:52.820569 19261 tablet.cc:2369] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:20:52.907975 19547 raft_consensus.cc:482] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:20:52.908325 19547 raft_consensus.cc:504] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 } } peers { permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 } } peers { permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 } }
I20260928 02:20:52.910158 19547 leader_election.cc:288] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e039622a54c24432a124a9c8decca201 (127.4.140.194:39979), 199fb8589bdb46b881841d23d59c85ce (127.4.140.193:44479)
I20260928 02:20:52.917397 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:52.911150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:47257 (local address 127.4.140.193:44479)
0928 02:20:52.911359 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:52.911364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:52.911384 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:20:52.911465 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:52.911470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:52.911563 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:52.911655 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:20:52.912141 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.913048 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.915401 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.916556 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.916713 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:52.916791 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:52.917015 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 02:20:52.917160 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:20:52.917404 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:52.910446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:44479 (local address 127.4.140.195:47257)
0928 02:20:52.910738 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:52.910759 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:52.910896 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:52.911705 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:52.911721 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:52.911752 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:20:52.911964 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:52.911979 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.913195 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.913199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:52.915193 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:52.915209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.916713 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.916717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:52.916857 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:52.916882 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:20:52.916959 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:20:52.917055 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260928 02:20:52.917994 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:52.910796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:48605 (local address 127.4.140.194:39979)
0928 02:20:52.911089 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:52.911098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:52.911598 (+ 500us) server_negotiation.cc:407] Connection header received
0928 02:20:52.911791 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:52.911799 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:52.911876 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:52.911987 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:20:52.912626 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.913517 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.915738 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.917001 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.917159 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:52.917227 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:52.917586 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 02:20:52.917739 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 02:20:52.918586 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:52.910478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:39979 (local address 127.4.140.195:48605)
0928 02:20:52.911467 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:52.911483 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:20:52.911620 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:20:52.912181 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:20:52.912184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:20:52.912202 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:20:52.912472 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:52.912479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.913678 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.913685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:52.915586 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:20:52.915596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:20:52.917205 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:20:52.917211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:20:52.917377 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:52.917420 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:20:52.918096 (+ 676us) client_negotiation.cc:240] Negotiation successful
0928 02:20:52.918272 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"thread_start_us":664,"threads_started":1}
I20260928 02:20:52.918605 19215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da613610d334ae5a0135d2b66d2898f" candidate_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "199fb8589bdb46b881841d23d59c85ce" is_pre_election: true
I20260928 02:20:52.919329 19215 raft_consensus.cc:2454] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d77e6699c98b4ca7b4065c17dfaa3e4c in term 0.
I20260928 02:20:52.919697 19347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da613610d334ae5a0135d2b66d2898f" candidate_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e039622a54c24432a124a9c8decca201" is_pre_election: true
I20260928 02:20:52.920126 19412 leader_election.cc:302] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 199fb8589bdb46b881841d23d59c85ce, d77e6699c98b4ca7b4065c17dfaa3e4c; no voters:
I20260928 02:20:52.920234 19347 raft_consensus.cc:2454] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d77e6699c98b4ca7b4065c17dfaa3e4c in term 0.
I20260928 02:20:52.920585 19547 raft_consensus.cc:2781] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:20:52.920769 19547 raft_consensus.cc:482] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:20:52.920915 19547 raft_consensus.cc:3037] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:20:52.923056 19547 raft_consensus.cc:504] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 } } peers { permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 } } peers { permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 } }
I20260928 02:20:52.923663 19547 leader_election.cc:288] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [CANDIDATE]: Term 1 election: Requested vote from peers e039622a54c24432a124a9c8decca201 (127.4.140.194:39979), 199fb8589bdb46b881841d23d59c85ce (127.4.140.193:44479)
I20260928 02:20:52.924188 19347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da613610d334ae5a0135d2b66d2898f" candidate_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e039622a54c24432a124a9c8decca201"
I20260928 02:20:52.924186 19215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da613610d334ae5a0135d2b66d2898f" candidate_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "199fb8589bdb46b881841d23d59c85ce"
I20260928 02:20:52.924400 19347 raft_consensus.cc:3037] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:20:52.924400 19215 raft_consensus.cc:3037] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:20:52.927646 19215 raft_consensus.cc:2454] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d77e6699c98b4ca7b4065c17dfaa3e4c in term 1.
I20260928 02:20:52.927646 19347 raft_consensus.cc:2454] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d77e6699c98b4ca7b4065c17dfaa3e4c in term 1.
I20260928 02:20:52.928191 19411 leader_election.cc:302] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d77e6699c98b4ca7b4065c17dfaa3e4c, e039622a54c24432a124a9c8decca201; no voters:
I20260928 02:20:52.928539 19547 raft_consensus.cc:2781] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:20:52.928951 19547 raft_consensus.cc:686] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [term 1 LEADER]: Becoming Leader. State: Replica: d77e6699c98b4ca7b4065c17dfaa3e4c, State: Running, Role: LEADER
I20260928 02:20:52.929539 19547 consensus_queue.cc:237] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 } } peers { permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 } } peers { permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 } }
I20260928 02:20:52.934091 19074 catalog_manager.cc:5495] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c reported cstate change: term changed from 0 to 1, leader changed from <none> to d77e6699c98b4ca7b4065c17dfaa3e4c (127.4.140.195). New cstate: current_term: 1 leader_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33023 } health_report { overall_health: HEALTHY } } }
I20260928 02:20:53.013566 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:53.007379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59444 (local address 127.4.140.195:33023)
0928 02:20:53.007619 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:53.007626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:53.007643 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:20:53.007745 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:53.007749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:53.007826 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:53.008006 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:20:53.008413 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:53.009214 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:53.010745 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:53.011321 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:53.011450 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:53.012072 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:20:53.012729 (+ 657us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:20:53.012925 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:53.013153 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 02:20:53.013250 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:20:53.022569 19215 raft_consensus.cc:1261] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce [term 1 FOLLOWER]: Refusing update from remote peer d77e6699c98b4ca7b4065c17dfaa3e4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:20:53.022599 19347 raft_consensus.cc:1261] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201 [term 1 FOLLOWER]: Refusing update from remote peer d77e6699c98b4ca7b4065c17dfaa3e4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:20:53.023538 19549 consensus_queue.cc:1035] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "199fb8589bdb46b881841d23d59c85ce" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:20:53.023957 19547 consensus_queue.cc:1035] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e039622a54c24432a124a9c8decca201" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:20:53.040793 19552 mvcc.cc:205] Tried to move back new op lower bound from 7334142169169620992 to 7334142168803688448. Current Snapshot: MvccSnapshot[applied={T|T < 7334142169169620992}]
I20260928 02:20:53.065982 19074 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:20:53.066262 19074 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:20:53.070456 19074 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c: Sending DeleteTablet for 3 replicas of tablet 9da613610d334ae5a0135d2b66d2898f
I20260928 02:20:53.072310 19195 tablet_service.cc:1514] Processing DeleteTablet for tablet 9da613610d334ae5a0135d2b66d2898f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:20:53 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:41388
I20260928 02:20:53.073071 19458 tablet_service.cc:1514] Processing DeleteTablet for tablet 9da613610d334ae5a0135d2b66d2898f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:20:53 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:59428
I20260928 02:20:53.073468 19562 tablet_replica.cc:330] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce: stopping tablet replica
I20260928 02:20:53.074062 19562 raft_consensus.cc:2229] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:20:53.074436 19327 tablet_service.cc:1514] Processing DeleteTablet for tablet 9da613610d334ae5a0135d2b66d2898f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:20:53 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:56996
I20260928 02:20:53.074980 19562 raft_consensus.cc:2258] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:20:53.075546 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 19112
I20260928 02:20:53.077852 19562 ts_tablet_manager.cc:1905] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:20:53.085103 19564 tablet_replica.cc:330] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201: stopping tablet replica
I20260928 02:20:53.085592 19562 ts_tablet_manager.cc:1918] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:20:53.085649 19564 raft_consensus.cc:2229] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:20:53.085875 19562 log.cc:1196] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/9da613610d334ae5a0135d2b66d2898f
I20260928 02:20:53.086249 19564 raft_consensus.cc:2258] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:20:53.086611 19562 ts_tablet_manager.cc:1939] T 9da613610d334ae5a0135d2b66d2898f P 199fb8589bdb46b881841d23d59c85ce: Deleting consensus metadata
I20260928 02:20:53.088514 19060 catalog_manager.cc:4841] TS 199fb8589bdb46b881841d23d59c85ce (127.4.140.193:44479): tablet 9da613610d334ae5a0135d2b66d2898f (table test-table [id=98d5790f060a4459937d010d0b59ef2f]) successfully deleted
I20260928 02:20:53.089404 19563 tablet_replica.cc:330] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c: stopping tablet replica
I20260928 02:20:53.089814 19563 raft_consensus.cc:2229] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:20:53.090510 19563 raft_consensus.cc:2258] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:20:53.092595 19564 ts_tablet_manager.cc:1905] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:20:53.093709 19563 ts_tablet_manager.cc:1905] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:20:53.098970 19564 ts_tablet_manager.cc:1918] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:20:53.099148 19564 log.cc:1196] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/9da613610d334ae5a0135d2b66d2898f
I20260928 02:20:53.099696 19564 ts_tablet_manager.cc:1939] T 9da613610d334ae5a0135d2b66d2898f P e039622a54c24432a124a9c8decca201: Deleting consensus metadata
I20260928 02:20:53.101600 19059 catalog_manager.cc:4841] TS e039622a54c24432a124a9c8decca201 (127.4.140.194:39979): tablet 9da613610d334ae5a0135d2b66d2898f (table test-table [id=98d5790f060a4459937d010d0b59ef2f]) successfully deleted
I20260928 02:20:53.103111 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:53.076632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41392 (local address 127.4.140.193:44479)
0928 02:20:53.076802 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:53.076808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:53.084492 (+ 7684us) server_negotiation.cc:407] Connection header received
0928 02:20:53.084699 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:53.084708 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:53.084819 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:53.084921 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:53.088948 (+ 4027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:53.090032 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:53.091934 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:53.092550 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:53.092672 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:53.096160 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:53.096176 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:53.096194 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:53.096244 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:53.099958 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:53.100745 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:53.100753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:53.100757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:53.100905 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:53.101327 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:53.101332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:53.101335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:53.101856 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:53.102175 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:53.102518 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 02:20:53.102757 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:20:53.103783 19255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:20:53.106076 19563 ts_tablet_manager.cc:1918] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:20:53.106266 19563 log.cc:1196] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/9da613610d334ae5a0135d2b66d2898f
I20260928 02:20:53.106894 19563 ts_tablet_manager.cc:1939] T 9da613610d334ae5a0135d2b66d2898f P d77e6699c98b4ca7b4065c17dfaa3e4c: Deleting consensus metadata
I20260928 02:20:53.108831 19060 catalog_manager.cc:4841] TS d77e6699c98b4ca7b4065c17dfaa3e4c (127.4.140.195:33023): tablet 9da613610d334ae5a0135d2b66d2898f (table test-table [id=98d5790f060a4459937d010d0b59ef2f]) successfully deleted
I20260928 02:20:53.193742 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 19112
I20260928 02:20:53.212770 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 19264
I20260928 02:20:53.229035 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:53.213639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57008 (local address 127.4.140.194:39979)
0928 02:20:53.213863 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:53.213870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:53.213893 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:20:53.214046 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:53.214052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:53.214145 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:53.214266 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:53.215555 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:53.216724 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:53.218143 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:53.218844 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:53.219003 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:53.221913 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:53.221931 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:53.221951 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:53.221996 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:53.225853 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:53.226582 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:53.226592 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:53.226596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:53.226773 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:53.227240 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:53.227246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:53.227249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:53.227722 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:53.228034 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:53.228381 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 02:20:53.228632 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:20:53.229745 19387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:20:53.320303 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 19264
I20260928 02:20:53.339098 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 19395
I20260928 02:20:53.355602 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:53.339954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.4.140.195:33023)
0928 02:20:53.340179 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:20:53.340186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:20:53.340341 (+ 155us) server_negotiation.cc:407] Connection header received
0928 02:20:53.340548 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:20:53.340552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:20:53.340621 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:20:53.340704 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:20:53.342586 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:53.343416 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:53.345403 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:20:53.345839 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:20:53.345935 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:20:53.349320 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:20:53.349332 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:20:53.349348 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:20:53.349389 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:20:53.352623 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:53.353507 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:53.353517 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:53.353519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:53.353666 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:20:53.354089 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:20:53.354092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:20:53.354094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:20:53.354430 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:20:53.354669 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 02:20:53.355042 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 02:20:53.355229 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:20:53.356336 19518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:20:53.479643 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 19395
I20260928 02:20:53.498912 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 19042
I20260928 02:20:53.500175 19105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:20:53.613559 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 19042
I20260928 02:20:53.635830 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:41423
--webserver_interface=127.4.140.254
--webserver_port=46673
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:41423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:20:53.966791 19570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:53.967090 19570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:53.967260 19570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:53.974516 19570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:20:53.974637 19570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:53.974684 19570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:53.974773 19570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:20:53.974843 19570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:20:53.984812 19570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:41423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:41423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=46673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:53.986195 19570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:53.987543 19570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:53.999585 19578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:53.999647 19576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:53.999615 19575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:54.000319 19570 server_base.cc:1034] running on GCE node
I20260928 02:20:54.001127 19570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:54.002588 19570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:54.003820 19570 hybrid_clock.cc:648] HybridClock initialized: now 1790562054003772 us; error 78 us; skew 500 ppm
I20260928 02:20:54.004216 19570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:54 dist-test-slave-h37r krb5kdc[18491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562054, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:54.007296 19570 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:20:54.008881 19570 webserver.cc:460] Webserver started at http://127.4.140.254:46673/ using document root <none> and password file <none>
I20260928 02:20:54.009588 19570 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:54.009742 19570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:20:54.015682 19570 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:20:54.019589 19585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:54.021142 19570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:20:54.021358 19570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "840ca23fcce445ac9d351c4f3a93e39c"
format_stamp: "Formatted at 2026-09-28 02:20:47 on dist-test-slave-h37r"
server_key: "f0552abebdc67fc16cd1228b0f54ec56"
server_key_iv: "ea528ebd4cb2b5cc07147056868fa098"
server_key_version: "encryptionkey@0"
I20260928 02:20:54.022428 19570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:54.039103 19570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:54.043792 19570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:54.044157 19570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:54.062507 19570 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:41423
I20260928 02:20:54.062609 19636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:41423 every 8 connection(s)
I20260928 02:20:54.064128 19570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:20:54.068797 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 19570
I20260928 02:20:54.069664 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:44479
--local_ip_for_outbound_sockets=127.4.140.193
--tserver_master_addrs=127.4.140.254:41423
--webserver_port=33891
--webserver_interface=127.4.140.193
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:20:54.071388 19637 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:20:54.074680 19637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c: Bootstrap starting.
I20260928 02:20:54.105350 19637 log.cc:824] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:20:54.128739 19637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:20:54.129403 19637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c: Bootstrap complete.
I20260928 02:20:54.138554 19637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ca23fcce445ac9d351c4f3a93e39c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 41423 } }
I20260928 02:20:54.139541 19637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 840ca23fcce445ac9d351c4f3a93e39c, State: Initialized, Role: FOLLOWER
I20260928 02:20:54.140224 19637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ca23fcce445ac9d351c4f3a93e39c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 41423 } }
I20260928 02:20:54.140470 19637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:20:54.140610 19637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:20:54.140825 19637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:20:54.144682 19637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ca23fcce445ac9d351c4f3a93e39c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 41423 } }
I20260928 02:20:54.145190 19637 leader_election.cc:302] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 840ca23fcce445ac9d351c4f3a93e39c; no voters:
I20260928 02:20:54.145805 19637 leader_election.cc:288] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 02:20:54.146020 19641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:20:54.147215 19641 raft_consensus.cc:686] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [term 3 LEADER]: Becoming Leader. State: Replica: 840ca23fcce445ac9d351c4f3a93e39c, State: Running, Role: LEADER
I20260928 02:20:54.147962 19637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:20:54.147830 19641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ca23fcce445ac9d351c4f3a93e39c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 41423 } }
I20260928 02:20:54.154398 19642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "840ca23fcce445ac9d351c4f3a93e39c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ca23fcce445ac9d351c4f3a93e39c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 41423 } } }
I20260928 02:20:54.154749 19642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [sys.catalog]: This master's current role is: LEADER
I20260928 02:20:54.155378 19643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 840ca23fcce445ac9d351c4f3a93e39c. Latest consensus state: current_term: 3 leader_uuid: "840ca23fcce445ac9d351c4f3a93e39c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ca23fcce445ac9d351c4f3a93e39c" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 41423 } } }
I20260928 02:20:54.155732 19643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840ca23fcce445ac9d351c4f3a93e39c [sys.catalog]: This master's current role is: LEADER
I20260928 02:20:54.162384 19657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:20:54.170430 19657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:20:54.171830 19657 catalog_manager.cc:1259] Loaded cluster ID: b705a640770c455285a04c1de807667b
I20260928 02:20:54.171964 19657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 02:20:54.173974 19657 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbc825dc9ad google::LogMessage::Fail() at ??:0
@ 0x7fbc825df88a google::LogMessage::SendToLog() at ??:0
@ 0x7fbc825dc4de google::LogMessage::Flush() at ??:0
@ 0x7fbc825dda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbc98013512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fbc98010976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbc98098610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fbc98098439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fbc85228345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbc8522b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fbc8522b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fbc85206eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbc884e06db start_thread at ??:0
@ 0x7fbc8147661f clone at ??:0
W20260928 02:20:54.481083 19639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:54.481353 19639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:54.481518 19639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:54.488718 19639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:54.488816 19639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:54.489017 19639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:20:54.499241 19639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:44479
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=33891
--tserver_master_addrs=127.4.140.254:41423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:54.500562 19639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:54.501888 19639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:54.514870 19681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:54.514948 19678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:54.514948 19679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:54.515750 19639 server_base.cc:1034] running on GCE node
I20260928 02:20:54.516376 19639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:54.517374 19639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:54.518630 19639 hybrid_clock.cc:648] HybridClock initialized: now 1790562054518582 us; error 58 us; skew 500 ppm
I20260928 02:20:54.518947 19639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:54 dist-test-slave-h37r krb5kdc[18491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562054, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:54.522341 19639 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:20:54.523746 19639 webserver.cc:460] Webserver started at http://127.4.140.193:33891/ using document root <none> and password file <none>
I20260928 02:20:54.524461 19639 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:54.524562 19639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:20:54.529673 19639 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:20:54.532361 19688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:54.533712 19639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:20:54.533885 19639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "199fb8589bdb46b881841d23d59c85ce"
format_stamp: "Formatted at 2026-09-28 02:20:48 on dist-test-slave-h37r"
server_key: "fcfdef9461a9bfebe90d259964ee0eb1"
server_key_iv: "439bb4699a94c721721230bdeed68aa4"
server_key_version: "encryptionkey@0"
I20260928 02:20:54.534866 19639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:54.559680 19639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:54.564116 19639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:54.564399 19639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:54.565438 19639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:20:54.566916 19639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:20:54.567051 19639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:54.567186 19639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:20:54.567276 19639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:54.601498 19639 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:44479
I20260928 02:20:54.601589 19800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:44479 every 8 connection(s)
I20260928 02:20:54.603011 19639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:20:54.608089 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 19639
I20260928 02:20:54.608374 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:54.606194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.193:52399)
0928 02:20:54.606892 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:54.607145 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":265,"threads_started":1}
I20260928 02:20:54.608876 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:39979
--local_ip_for_outbound_sockets=127.4.140.194
--tserver_master_addrs=127.4.140.254:41423
--webserver_port=46133
--webserver_interface=127.4.140.194
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:20:54.609138 19801 heartbeater.cc:644] Failed to heartbeat to 127.4.140.254:41423 (0 consecutive failures): Network error: Failed to ping master at 127.4.140.254:41423: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
I20260928 02:20:54.610157 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:54.609698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.193:57007)
0928 02:20:54.609900 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:54.609999 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 02:20:54.611196 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:54.610780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.193:52867)
0928 02:20:54.610948 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:54.611044 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 02:20:54.611491 19801 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:20:54.938643 19804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:54.938957 19804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:54.939136 19804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:54.946511 19804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:54.946612 19804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:54.946722 19804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:20:54.956336 19804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:39979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=46133
--tserver_master_addrs=127.4.140.254:41423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:54.957597 19804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:54.958915 19804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:54.972167 19809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:54.972239 19810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:54.972281 19812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:54.973613 19804 server_base.cc:1034] running on GCE node
I20260928 02:20:54.974278 19804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:54.975291 19804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:54.976521 19804 hybrid_clock.cc:648] HybridClock initialized: now 1790562054976473 us; error 73 us; skew 500 ppm
I20260928 02:20:54.976842 19804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:54 dist-test-slave-h37r krb5kdc[18491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562054, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:54.979861 19804 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:20:54.981457 19804 webserver.cc:460] Webserver started at http://127.4.140.194:46133/ using document root <none> and password file <none>
I20260928 02:20:54.982187 19804 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:54.982334 19804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:20:54.987584 19804 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:20:54.990300 19819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:54.991739 19804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:20:54.991981 19804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "e039622a54c24432a124a9c8decca201"
format_stamp: "Formatted at 2026-09-28 02:20:49 on dist-test-slave-h37r"
server_key: "9490986a5277a8a2b9281d368562b6a9"
server_key_iv: "e253371e7022cfd984f1f05c4156b4b6"
server_key_version: "encryptionkey@0"
I20260928 02:20:54.992983 19804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:55.014930 19804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:55.019528 19804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:55.019865 19804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:55.021709 19804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:20:55.023102 19804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:20:55.023234 19804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:55.023370 19804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:20:55.023468 19804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:55.057193 19804 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:39979
I20260928 02:20:55.057257 19931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:39979 every 8 connection(s)
I20260928 02:20:55.058730 19804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:20:55.062404 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 19804
I20260928 02:20:55.063170 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:33023
--local_ip_for_outbound_sockets=127.4.140.195
--tserver_master_addrs=127.4.140.254:41423
--webserver_port=35817
--webserver_interface=127.4.140.195
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:20:55.064044 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:55.061827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.194:44783)
0928 02:20:55.062700 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:55.062973 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":346,"threads_started":1}
W20260928 02:20:55.064750 19932 heartbeater.cc:644] Failed to heartbeat to 127.4.140.254:41423 (0 consecutive failures): Network error: Failed to ping master at 127.4.140.254:41423: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
I20260928 02:20:55.065718 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:55.065400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.194:53933)
0928 02:20:55.065549 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:55.065618 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 02:20:55.066787 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:55.066291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.194:52881)
0928 02:20:55.066488 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:55.066603 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 02:20:55.067166 19932 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:20:55.384413 19935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:20:55.384780 19935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:20:55.384946 19935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:20:55.392234 19935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:20:55.392383 19935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:20:55.392556 19935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:20:55.402328 19935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:33023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=35817
--tserver_master_addrs=127.4.140.254:41423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:20:55.403656 19935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:20:55.405004 19935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:20:55.418119 19941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:55.418136 19943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:20:55.418161 19940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:20:55.419191 19935 server_base.cc:1034] running on GCE node
I20260928 02:20:55.419782 19935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:20:55.420837 19935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:20:55.422031 19935 hybrid_clock.cc:648] HybridClock initialized: now 1790562055421999 us; error 30 us; skew 500 ppm
I20260928 02:20:55.422348 19935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:20:55 dist-test-slave-h37r krb5kdc[18491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562055, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:20:55.425369 19935 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:20:55.426772 19935 webserver.cc:460] Webserver started at http://127.4.140.195:35817/ using document root <none> and password file <none>
I20260928 02:20:55.427505 19935 fs_manager.cc:359] Metadata directory not provided
I20260928 02:20:55.427651 19935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:20:55.432897 19935 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 02:20:55.435622 19950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:55.437024 19935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:20:55.437218 19935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "d77e6699c98b4ca7b4065c17dfaa3e4c"
format_stamp: "Formatted at 2026-09-28 02:20:49 on dist-test-slave-h37r"
server_key: "16cdc532f6110b1cddc6138989626516"
server_key_iv: "9e0fbf56a5a8734d57d9984faa4026e6"
server_key_version: "encryptionkey@0"
I20260928 02:20:55.438237 19935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:20:55.453377 19935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:20:55.457576 19935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:20:55.457863 19935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:20:55.458839 19935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:20:55.460264 19935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:20:55.460386 19935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:55.460520 19935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:20:55.460609 19935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:20:55.493736 19935 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:33023
I20260928 02:20:55.493821 20062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:33023 every 8 connection(s)
I20260928 02:20:55.495229 19935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:20:55.497294 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 19935
I20260928 02:20:55.500494 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:55.498512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.195:51801)
0928 02:20:55.499358 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:55.499582 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":314,"threads_started":1}
W20260928 02:20:55.501194 20063 heartbeater.cc:644] Failed to heartbeat to 127.4.140.254:41423 (0 consecutive failures): Network error: Failed to ping master at 127.4.140.254:41423: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
I20260928 02:20:55.501971 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:55.501670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.195:57507)
0928 02:20:55.501806 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:55.501871 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 02:20:55.502774 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:55.502488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.195:49253)
0928 02:20:55.502598 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:55.502665 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 02:20:55.503052 20063 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:20:55.612885 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:55.612121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.193:44225)
0928 02:20:55.612575 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:55.612659 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":225,"threads_started":1}
I20260928 02:20:56.068434 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:56.067629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.194:59831)
0928 02:20:56.068128 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:56.068210 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":226,"threads_started":1}
I20260928 02:20:56.504527 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:56.503674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.195:43479)
0928 02:20:56.504167 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:56.504298 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":272,"threads_started":1}
I20260928 02:20:56.614552 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:56.613836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.193:58387)
0928 02:20:56.614301 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:56.614381 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":242,"threads_started":1}
I20260928 02:20:57.070147 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:57.069399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.194:36139)
0928 02:20:57.069883 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:57.069974 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":226,"threads_started":1}
I20260928 02:20:57.506110 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:57.505427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.195:33377)
0928 02:20:57.505864 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:57.505946 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":213,"threads_started":1}
I20260928 02:20:57.616519 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:57.615567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.193:60107)
0928 02:20:57.616159 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:57.616258 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":320,"threads_started":1}
I20260928 02:20:58.072060 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:58.071186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.194:38979)
0928 02:20:58.071702 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:58.071810 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":231,"threads_started":1}
I20260928 02:20:58.507826 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:58.507013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.195:60543)
0928 02:20:58.507549 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:58.507650 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":328,"threads_started":1}
I20260928 02:20:58.618317 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:58.617549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.193:42105)
0928 02:20:58.618057 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:58.618146 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":268,"threads_started":1}
I20260928 02:20:59.073662 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:59.072915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.194:40067)
0928 02:20:59.073361 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:59.073460 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":222,"threads_started":1}
I20260928 02:20:59.509456 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:59.508773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.195:34661)
0928 02:20:59.509186 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:59.509264 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":221,"threads_started":1}
I20260928 02:20:59.620260 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:20:59.619339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.193:49427)
0928 02:20:59.619991 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 02:20:59.620075 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":359,"threads_started":1}
I20260928 02:21:00.075470 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:00.074719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.194:48407)
0928 02:21:00.075200 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:00.075295 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":234,"threads_started":1}
I20260928 02:21:00.511318 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:00.510560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.195:38903)
0928 02:21:00.511042 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:00.511129 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":255,"threads_started":1}
I20260928 02:21:00.621979 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:00.621170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.193:37989)
0928 02:21:00.621706 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:00.621810 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":256,"threads_started":1}
I20260928 02:21:01.077344 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:01.076611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.194:45759)
0928 02:21:01.077065 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:01.077158 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":236,"threads_started":1}
I20260928 02:21:01.513109 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:01.512388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.195:48051)
0928 02:21:01.512842 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:01.512941 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":233,"threads_started":1}
I20260928 02:21:01.623804 20085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:01.623001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.193:41967)
0928 02:21:01.623500 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:01.623589 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":227,"threads_started":1}
I20260928 02:21:02.080629 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:02.078450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.194:38163)
0928 02:21:02.080319 (+ 1869us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:02.080399 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1751,"thread_start_us":1582,"threads_started":1}
I20260928 02:21:02.515218 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:02.514316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.195:50129)
0928 02:21:02.514900 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:02.514980 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":276,"threads_started":1}
I20260928 02:21:02.625689 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:02.624733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.193:51933)
0928 02:21:02.625353 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:02.625443 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":326,"threads_started":1}
I20260928 02:21:03.082296 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:03.081611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.194:47429)
0928 02:21:03.082043 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:03.082122 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":219,"threads_started":1}
I20260928 02:21:03.516980 20090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:03.516244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.195:43171)
0928 02:21:03.516724 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:03.516802 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":268,"threads_started":1}
I20260928 02:21:03.627552 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:03.626715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.193:37283)
0928 02:21:03.627183 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:03.627295 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":229,"threads_started":1}
I20260928 02:21:04.084232 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:04.083404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.194:52055)
0928 02:21:04.083875 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:04.084043 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":223,"threads_started":1}
I20260928 02:21:04.518754 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:04.517893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.195:46189)
0928 02:21:04.518483 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:04.518579 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":322,"threads_started":1}
I20260928 02:21:04.629384 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:04.628691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.193:51121)
0928 02:21:04.629123 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:04.629223 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":237,"threads_started":1}
I20260928 02:21:05.085930 20095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:05.085177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.194:38275)
0928 02:21:05.085679 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:05.085769 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":260,"threads_started":1}
I20260928 02:21:05.520481 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:05.519708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.195:59025)
0928 02:21:05.520232 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:05.520317 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":316,"threads_started":1}
I20260928 02:21:05.631124 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:05.630368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.193:59141)
0928 02:21:05.630844 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:05.630942 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":228,"threads_started":1}
I20260928 02:21:06.087734 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:06.087001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.194:59483)
0928 02:21:06.087453 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:06.087554 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":231,"threads_started":1}
I20260928 02:21:06.522415 20099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:06.521609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.195:49725)
0928 02:21:06.522135 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:06.522232 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":267,"threads_started":1}
I20260928 02:21:06.632896 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:06.632199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.193:35535)
0928 02:21:06.632649 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:06.632737 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":233,"threads_started":1}
I20260928 02:21:07.089488 20101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:07.088773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.194:42643)
0928 02:21:07.089240 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:07.089326 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":241,"threads_started":1}
I20260928 02:21:07.524312 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:07.523542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.195:43143)
0928 02:21:07.524042 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:07.524129 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":226,"threads_started":1}
I20260928 02:21:07.634703 20103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:07.633882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.193:40531)
0928 02:21:07.634406 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:07.634509 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":256,"threads_started":1}
I20260928 02:21:08.091250 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:08.090464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.194:56681)
0928 02:21:08.090951 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:08.091044 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":237,"threads_started":1}
I20260928 02:21:08.526031 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:08.525270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.195:52615)
0928 02:21:08.525748 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:08.525845 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":238,"threads_started":1}
I20260928 02:21:08.636461 20106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:08.635673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.193:34681)
0928 02:21:08.636184 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:08.636273 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":296,"threads_started":1}
I20260928 02:21:09.093123 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:09.092320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.194:37671)
0928 02:21:09.092843 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:09.092926 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":300,"threads_started":1}
I20260928 02:21:09.527923 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:09.526988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.195:58373)
0928 02:21:09.527533 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:09.527655 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":234,"threads_started":1}
I20260928 02:21:09.638458 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:09.637671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.193:53863)
0928 02:21:09.638143 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:09.638251 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":237,"threads_started":1}
I20260928 02:21:10.095080 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:10.094271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.194:36641)
0928 02:21:10.094769 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:10.094885 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":277,"threads_started":1}
I20260928 02:21:10.505056 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 19639
I20260928 02:21:10.529570 20111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:10.506255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38152 (local address 127.4.140.193:44479)
0928 02:21:10.506825 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:10.506846 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:10.507078 (+ 232us) server_negotiation.cc:407] Connection header received
0928 02:21:10.508820 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:10.508830 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:10.509284 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:10.509662 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:21:10.511182 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:10.512541 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:10.513842 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:10.514534 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:10.514747 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:10.522578 (+ 7831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:21:10.522588 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:21:10.522606 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:21:10.522650 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:21:10.525865 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:10.526626 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:10.526641 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:10.526647 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:10.526826 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:10.527238 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:10.527243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:10.527246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:10.527802 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:21:10.528185 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:10.528686 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 02:21:10.529264 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":311,"threads_started":1}
I20260928 02:21:10.530114 20112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:10.529166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:41423 (local address 127.4.140.195:45943)
0928 02:21:10.529758 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:10.529883 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":311,"threads_started":1}
I20260928 02:21:10.531347 19796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:21:10.618563 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 19639
I20260928 02:21:10.637964 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 19804
I20260928 02:21:10.656641 20114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:10.639129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.4.140.194:39979)
0928 02:21:10.639721 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:10.639736 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:10.639774 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:21:10.641426 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:10.641436 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:10.641959 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:10.642278 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:21:10.643673 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:10.645081 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:10.646292 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:10.646934 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:10.647072 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:10.650104 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:21:10.650111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:21:10.650126 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:21:10.650162 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:21:10.653145 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:10.654014 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:10.654020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:10.654027 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:10.654187 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:10.654596 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:10.654599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:10.654601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:10.655098 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:21:10.655358 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:10.655970 (+ 612us) server_negotiation.cc:299] Negotiation successful
0928 02:21:10.656353 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":299,"threads_started":1}
I20260928 02:21:10.658066 19927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:21:10.741631 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 19804
I20260928 02:21:10.761683 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 19935
I20260928 02:21:10.781293 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:10.762711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48146 (local address 127.4.140.195:33023)
0928 02:21:10.763391 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:10.763406 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:10.763449 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:21:10.765139 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:10.765149 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:10.765611 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:10.765905 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:21:10.767296 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:10.769014 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:10.770271 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:10.771029 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:10.771219 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:10.773981 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:21:10.773991 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:21:10.774008 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:21:10.774058 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:21:10.777522 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:10.778299 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:10.778307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:10.778314 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:10.778536 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:10.779024 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:10.779028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:10.779030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:10.779668 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:21:10.780054 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:10.780542 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 02:21:10.780916 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":333,"threads_started":1}
I20260928 02:21:10.782868 20058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:21:10.869958 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 19935
2026-09-28T02:21:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24644 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:21:10 dist-test-slave-h37r krb5kdc[20120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:21:10 dist-test-slave-h37r krb5kdc[20120](info): set up 2 sockets
Sep 28 02:21:10 dist-test-slave-h37r krb5kdc[20120](info): commencing operation
krb5kdc: starting...
W20260928 02:21:11.976769 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:21:12 dist-test-slave-h37r krb5kdc[20120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:21:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:21:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:21:12.245370 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:32967
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:32967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:21:12.580274 20133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:21:12.580549 20133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:21:12.580734 20133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:21:12.588030 20133 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:21:12.588130 20133 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:21:12.588219 20133 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:21:12.588327 20133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:21:12.588428 20133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:21:12.588534 20133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:21:12.588624 20133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:21:12.588706 20133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:21:12.598923 20133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:32967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:32967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:21:12.600296 20133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:21:12.601631 20133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:21:12.614347 20141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:12.614405 20139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:12.614364 20138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:21:12.614567 20133 server_base.cc:1034] running on GCE node
I20260928 02:21:12.615648 20133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:21:12.617101 20133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:21:12.618419 20133 hybrid_clock.cc:648] HybridClock initialized: now 1790562072618353 us; error 52 us; skew 500 ppm
I20260928 02:21:12.618911 20133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:21:12 dist-test-slave-h37r krb5kdc[20120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562072, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:21:12.622239 20133 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:21:12.623781 20133 webserver.cc:460] Webserver started at http://127.4.140.254:40115/ using document root <none> and password file <none>
I20260928 02:21:12.624542 20133 fs_manager.cc:359] Metadata directory not provided
I20260928 02:21:12.624660 20133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:21:12.624989 20133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:21:12.628042 20133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "6f3cfae4688a4df3bce475f9158ac2bb"
format_stamp: "Formatted at 2026-09-28 02:21:12 on dist-test-slave-h37r"
server_key: "2191e889c53695688ed10ac79da3fc5c"
server_key_iv: "2e0ad528d61e3705115c7c170c3ced9e"
server_key_version: "encryptionkey@0"
I20260928 02:21:12.628875 20133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "6f3cfae4688a4df3bce475f9158ac2bb"
format_stamp: "Formatted at 2026-09-28 02:21:12 on dist-test-slave-h37r"
server_key: "2191e889c53695688ed10ac79da3fc5c"
server_key_iv: "2e0ad528d61e3705115c7c170c3ced9e"
server_key_version: "encryptionkey@0"
I20260928 02:21:12.634012 20133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 02:21:12.637603 20148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:12.638937 20133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:21:12.639117 20133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "6f3cfae4688a4df3bce475f9158ac2bb"
format_stamp: "Formatted at 2026-09-28 02:21:12 on dist-test-slave-h37r"
server_key: "2191e889c53695688ed10ac79da3fc5c"
server_key_iv: "2e0ad528d61e3705115c7c170c3ced9e"
server_key_version: "encryptionkey@0"
I20260928 02:21:12.639355 20133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:21:12.659062 20133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:21:12.663527 20133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:21:12.663827 20133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:21:12.682749 20133 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:32967
I20260928 02:21:12.682767 20199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:32967 every 8 connection(s)
I20260928 02:21:12.684136 20133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:21:12.687902 20200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:21:12.689162 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 20133
I20260928 02:21:12.689417 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:21:12.690115 4659 external_mini_cluster.cc:1428] Setting key 0bbbc2a3ef1cbf42a4fb20edb789d676
I20260928 02:21:12.702270 20200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb: Bootstrap starting.
Sep 28 02:21:12 dist-test-slave-h37r krb5kdc[20120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:21:12.706030 20200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb: Neither blocks nor log segments found. Creating new log.
I20260928 02:21:12.707082 20200 log.cc:824] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:21:12.710067 20200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb: No bootstrap required, opened a new log
I20260928 02:21:12.713893 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:12.692181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.4.140.254:32967)
0928 02:21:12.693036 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:12.693051 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:12.693096 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:21:12.694660 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:12.694684 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:12.695094 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:12.695444 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:21:12.697267 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:12.698594 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:12.700544 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:12.701199 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:12.701323 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:12.706084 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:21:12.706122 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:21:12.706141 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:21:12.706203 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:21:12.709592 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:12.710478 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:12.710484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:12.710493 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:12.710664 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:12.711121 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:12.711130 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:12.711133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:12.711648 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:21:12.712030 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:12.712475 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 02:21:12.712939 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":620,"thread_start_us":403,"threads_started":1}
I20260928 02:21:12.717984 20200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32967 } }
I20260928 02:21:12.718350 20200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:21:12.718459 20200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3cfae4688a4df3bce475f9158ac2bb, State: Initialized, Role: FOLLOWER
I20260928 02:21:12.718972 20200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32967 } }
I20260928 02:21:12.719259 20200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:21:12.719395 20200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:21:12.719584 20200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:21:12.722107 20200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32967 } }
I20260928 02:21:12.722600 20200 leader_election.cc:302] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f3cfae4688a4df3bce475f9158ac2bb; no voters:
I20260928 02:21:12.723168 20200 leader_election.cc:288] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:21:12.723323 20205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:21:12.724617 20205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 1 LEADER]: Becoming Leader. State: Replica: 6f3cfae4688a4df3bce475f9158ac2bb, State: Running, Role: LEADER
I20260928 02:21:12.725108 20205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32967 } }
I20260928 02:21:12.725431 20200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:21:12.727965 20206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32967 } } }
I20260928 02:21:12.728144 20207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f3cfae4688a4df3bce475f9158ac2bb. Latest consensus state: current_term: 1 leader_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32967 } } }
I20260928 02:21:12.728459 20207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [sys.catalog]: This master's current role is: LEADER
I20260928 02:21:12.728467 20206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [sys.catalog]: This master's current role is: LEADER
I20260928 02:21:12.730314 20213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:21:12.735818 20213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:21:12.745050 20213 catalog_manager.cc:1347] Generated new cluster ID: 1720ff3f24594a0f95df1c88a5911f1d
I20260928 02:21:12.745182 20213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:21:12.765275 20213 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:21:12.767277 20213 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:21:12.776873 20213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb: Generated new TSK 0
I20260928 02:21:12.777733 20213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
I20260928 02:21:12.844913 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:32967
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:21:13.213137 20226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:21:13.213469 20226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:21:13.213649 20226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:21:13.220866 20226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:21:13.220988 20226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:21:13.221176 20226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:21:13.231467 20226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:32967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:21:13.232764 20226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:21:13.234094 20226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:21:13.247253 20231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:13.247327 20234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:13.247359 20232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:21:13.248011 20226 server_base.cc:1034] running on GCE node
I20260928 02:21:13.248602 20226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:21:13.249454 20226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:21:13.250712 20226 hybrid_clock.cc:648] HybridClock initialized: now 1790562073250667 us; error 61 us; skew 500 ppm
I20260928 02:21:13.251053 20226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:21:13 dist-test-slave-h37r krb5kdc[20120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562073, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:21:13.254231 20226 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:21:13.255681 20226 webserver.cc:460] Webserver started at http://127.4.140.193:42207/ using document root <none> and password file <none>
I20260928 02:21:13.256381 20226 fs_manager.cc:359] Metadata directory not provided
I20260928 02:21:13.256479 20226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:21:13.256755 20226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:21:13.259567 20226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "dd09ff4c20d44abab01c6570962c8df8"
format_stamp: "Formatted at 2026-09-28 02:21:13 on dist-test-slave-h37r"
server_key: "1362e6cc326596ae57af4dbad5196012"
server_key_iv: "77206a69e16a1db68e35e9cbafd99098"
server_key_version: "encryptionkey@0"
I20260928 02:21:13.260330 20226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "dd09ff4c20d44abab01c6570962c8df8"
format_stamp: "Formatted at 2026-09-28 02:21:13 on dist-test-slave-h37r"
server_key: "1362e6cc326596ae57af4dbad5196012"
server_key_iv: "77206a69e16a1db68e35e9cbafd99098"
server_key_version: "encryptionkey@0"
I20260928 02:21:13.265313 20226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 02:21:13.268899 20241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:13.270447 20226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:21:13.270640 20226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "dd09ff4c20d44abab01c6570962c8df8"
format_stamp: "Formatted at 2026-09-28 02:21:13 on dist-test-slave-h37r"
server_key: "1362e6cc326596ae57af4dbad5196012"
server_key_iv: "77206a69e16a1db68e35e9cbafd99098"
server_key_version: "encryptionkey@0"
I20260928 02:21:13.270844 20226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:21:13.283475 20226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:21:13.287822 20226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:21:13.288188 20226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:21:13.289134 20226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:21:13.290442 20226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:21:13.290512 20226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:13.290699 20226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:21:13.290763 20226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:13.324896 20226 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:39591
I20260928 02:21:13.324935 20353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:39591 every 8 connection(s)
I20260928 02:21:13.326273 20226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:21:13.332484 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 20226
I20260928 02:21:13.332727 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:21:13.333418 4659 external_mini_cluster.cc:1428] Setting key 3948cce6184fbc847d856790ff334a38
Sep 28 02:21:13 dist-test-slave-h37r krb5kdc[20120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562073, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:21:13.348205 20356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:13.329105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:38177 (local address 127.4.140.254:32967)
0928 02:21:13.329626 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:13.329630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:13.330623 (+ 993us) server_negotiation.cc:407] Connection header received
0928 02:21:13.332599 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:13.332605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:13.332715 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:13.332847 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:21:13.334428 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:13.335487 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:13.337140 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:13.337546 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:13.337707 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:13.341260 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:21:13.341265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:21:13.341268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:21:13.341300 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:21:13.344558 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:13.345565 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:13.345570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:13.345573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:13.345723 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:13.346244 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:13.346248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:13.346250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:13.346556 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:21:13.346680 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:13.347639 (+ 959us) server_negotiation.cc:299] Negotiation successful
0928 02:21:13.347819 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":295,"threads_started":1}
I20260928 02:21:13.349566 20357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:13.329677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.193:38177)
0928 02:21:13.330454 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:13.330498 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:21:13.332322 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:21:13.333083 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:21:13.333091 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:21:13.333641 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:21:13.334241 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:13.334260 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:13.335669 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:13.335674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:13.336983 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:13.336996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:13.337788 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:13.337797 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:13.338046 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:13.338827 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:21:13.338850 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:21:13.341024 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:21:13.344709 (+ 3685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:21:13.344722 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:21:13.344742 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:21:13.345398 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:21:13.345842 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:21:13.345850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:21:13.345856 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:21:13.346112 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:21:13.346701 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:21:13.346712 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:21:13.347179 (+ 467us) client_negotiation.cc:769] Sending connection context
0928 02:21:13.347489 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 02:21:13.348106 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":298,"threads_started":1}
I20260928 02:21:13.351301 20354 heartbeater.cc:347] Connected to a master server at 127.4.140.254:32967
I20260928 02:21:13.351706 20354 heartbeater.cc:464] Registering TS with master...
I20260928 02:21:13.352519 20354 heartbeater.cc:511] Master 127.4.140.254:32967 requested a full tablet report, sending...
I20260928 02:21:13.354835 20165 ts_manager.cc:194] Registered new tserver with Master: dd09ff4c20d44abab01c6570962c8df8 (127.4.140.193:39591)
I20260928 02:21:13.356432 20165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:38177
WARNING: no policy specified for kudu/127.4.140.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
I20260928 02:21:13.391199 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:32967
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:21:13.715824 20360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:21:13.716209 20360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:21:13.716395 20360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:21:13.723526 20360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:21:13.723709 20360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:21:13.723935 20360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:21:13.733836 20360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:32967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:21:13.735128 20360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:21:13.736542 20360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:21:13.749696 20365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:13.749737 20368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:13.749771 20366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:21:13.750370 20360 server_base.cc:1034] running on GCE node
I20260928 02:21:13.750988 20360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:21:13.752010 20360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:21:13.753443 20360 hybrid_clock.cc:648] HybridClock initialized: now 1790562073753199 us; error 31 us; skew 500 ppm
I20260928 02:21:13.753804 20360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:21:13 dist-test-slave-h37r krb5kdc[20120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562073, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:21:13.757050 20360 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:21:13.758477 20360 webserver.cc:460] Webserver started at http://127.4.140.194:37725/ using document root <none> and password file <none>
I20260928 02:21:13.759091 20360 fs_manager.cc:359] Metadata directory not provided
I20260928 02:21:13.759194 20360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:21:13.759451 20360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:21:13.762302 20360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "7bb12dd2a4d14e1f9d9470a75e58e031"
format_stamp: "Formatted at 2026-09-28 02:21:13 on dist-test-slave-h37r"
server_key: "7b89d87cd2c3076431081e3ffb1ffc8f"
server_key_iv: "91ed23319b8c1a56f68a792ccfd25f19"
server_key_version: "encryptionkey@0"
I20260928 02:21:13.763052 20360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "7bb12dd2a4d14e1f9d9470a75e58e031"
format_stamp: "Formatted at 2026-09-28 02:21:13 on dist-test-slave-h37r"
server_key: "7b89d87cd2c3076431081e3ffb1ffc8f"
server_key_iv: "91ed23319b8c1a56f68a792ccfd25f19"
server_key_version: "encryptionkey@0"
I20260928 02:21:13.768123 20360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260928 02:21:13.771530 20375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:13.772995 20360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:21:13.773190 20360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "7bb12dd2a4d14e1f9d9470a75e58e031"
format_stamp: "Formatted at 2026-09-28 02:21:13 on dist-test-slave-h37r"
server_key: "7b89d87cd2c3076431081e3ffb1ffc8f"
server_key_iv: "91ed23319b8c1a56f68a792ccfd25f19"
server_key_version: "encryptionkey@0"
I20260928 02:21:13.773396 20360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:21:13.795748 20360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:21:13.800045 20360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:21:13.800339 20360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:21:13.801373 20360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:21:13.802759 20360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:21:13.802876 20360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:13.803040 20360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:21:13.803119 20360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:13.839701 20360 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:37981
I20260928 02:21:13.839756 20487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:37981 every 8 connection(s)
I20260928 02:21:13.841178 20360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:21:13.844667 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 20360
I20260928 02:21:13.844863 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:21:13.845510 4659 external_mini_cluster.cc:1428] Setting key 51a3f256f8e92d4e1b223415d135d6a5
Sep 28 02:21:13 dist-test-slave-h37r krb5kdc[20120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562073, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:21:13.863298 20356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:13.844163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:55025 (local address 127.4.140.254:32967)
0928 02:21:13.844352 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:13.844356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:13.845959 (+ 1603us) server_negotiation.cc:407] Connection header received
0928 02:21:13.848541 (+ 2582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:13.848557 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:13.848647 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:13.848769 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:21:13.850335 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:13.851042 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:13.852700 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:13.853151 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:13.853297 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:13.857219 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:21:13.857224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:21:13.857227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:21:13.857259 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:21:13.860150 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:13.860930 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:13.860935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:13.860939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:13.861072 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:13.861535 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:13.861544 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:13.861546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:13.861814 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:21:13.861982 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:13.862700 (+ 718us) server_negotiation.cc:299] Negotiation successful
0928 02:21:13.862934 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:21:13.864442 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:13.844812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.194:55025)
0928 02:21:13.845766 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:13.845819 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:21:13.848269 (+ 2450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:21:13.849020 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:21:13.849028 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:21:13.849542 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:21:13.850149 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:13.850171 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:13.851211 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:13.851216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:13.852547 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:13.852560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:13.853386 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:13.853393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:13.853630 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:13.854334 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:21:13.854357 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:21:13.856961 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:21:13.860307 (+ 3346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:21:13.860317 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:21:13.860335 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:21:13.860785 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:21:13.861194 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:21:13.861200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:21:13.861204 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:21:13.861397 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:21:13.861953 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:21:13.861961 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:21:13.862321 (+ 360us) client_negotiation.cc:769] Sending connection context
0928 02:21:13.862600 (+ 279us) client_negotiation.cc:240] Negotiation successful
0928 02:21:13.863181 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":342,"threads_started":1}
I20260928 02:21:13.866163 20488 heartbeater.cc:347] Connected to a master server at 127.4.140.254:32967
I20260928 02:21:13.866510 20488 heartbeater.cc:464] Registering TS with master...
I20260928 02:21:13.867298 20488 heartbeater.cc:511] Master 127.4.140.254:32967 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.140.195@KRBTEST.COM; defaulting to no policy
I20260928 02:21:13.869065 20165 ts_manager.cc:194] Registered new tserver with Master: 7bb12dd2a4d14e1f9d9470a75e58e031 (127.4.140.194:37981)
I20260928 02:21:13.869959 20165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:55025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
I20260928 02:21:13.903209 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:32967
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:21:14.232740 20493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:21:14.233053 20493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:21:14.233232 20493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:21:14.240315 20493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:21:14.240469 20493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:21:14.240659 20493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:21:14.250401 20493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:32967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:21:14.251673 20493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:21:14.253000 20493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:21:14.266656 20498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:14.266546 20499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:14.266628 20501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:21:14.269547 20493 server_base.cc:1034] running on GCE node
I20260928 02:21:14.270236 20493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:21:14.271342 20493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:21:14.272601 20493 hybrid_clock.cc:648] HybridClock initialized: now 1790562074272546 us; error 38 us; skew 500 ppm
I20260928 02:21:14.272936 20493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:21:14 dist-test-slave-h37r krb5kdc[20120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562074, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:21:14.276444 20493 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:21:14.278108 20493 webserver.cc:460] Webserver started at http://127.4.140.195:43961/ using document root <none> and password file <none>
I20260928 02:21:14.278841 20493 fs_manager.cc:359] Metadata directory not provided
I20260928 02:21:14.278941 20493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:21:14.279193 20493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:21:14.282096 20493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "3e51bb23a4dc4b89ae61691398347cc5"
format_stamp: "Formatted at 2026-09-28 02:21:14 on dist-test-slave-h37r"
server_key: "2282be1320dfbfa766f8d74f257ef412"
server_key_iv: "cf8591c557f40ffdca81db044c199cc9"
server_key_version: "encryptionkey@0"
I20260928 02:21:14.282801 20493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "3e51bb23a4dc4b89ae61691398347cc5"
format_stamp: "Formatted at 2026-09-28 02:21:14 on dist-test-slave-h37r"
server_key: "2282be1320dfbfa766f8d74f257ef412"
server_key_iv: "cf8591c557f40ffdca81db044c199cc9"
server_key_version: "encryptionkey@0"
I20260928 02:21:14.287850 20493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:21:14.291275 20508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:14.292651 20493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:21:14.292790 20493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "3e51bb23a4dc4b89ae61691398347cc5"
format_stamp: "Formatted at 2026-09-28 02:21:14 on dist-test-slave-h37r"
server_key: "2282be1320dfbfa766f8d74f257ef412"
server_key_iv: "cf8591c557f40ffdca81db044c199cc9"
server_key_version: "encryptionkey@0"
I20260928 02:21:14.292958 20493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:21:14.314749 20493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:21:14.318893 20493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:21:14.319133 20493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:21:14.320127 20493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:21:14.321381 20493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:21:14.321455 20493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:14.321537 20493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:21:14.321579 20493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:14.357736 20493 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:44857
I20260928 02:21:14.357857 20620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:44857 every 8 connection(s)
I20260928 02:21:14.359064 20493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:21:14.359480 20354 heartbeater.cc:503] Master 127.4.140.254:32967 was elected leader, sending a full tablet report...
I20260928 02:21:14.367694 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 20493
I20260928 02:21:14.367923 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:21:14.368491 4659 external_mini_cluster.cc:1428] Setting key 08a894390af5958d4cd2fd650f54de38
Sep 28 02:21:14 dist-test-slave-h37r krb5kdc[20120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562074, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:21:14.379885 20356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:14.361845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:57785 (local address 127.4.140.254:32967)
0928 02:21:14.362054 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:14.362058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:14.363315 (+ 1257us) server_negotiation.cc:407] Connection header received
0928 02:21:14.365247 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:14.365251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:14.365338 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:14.365425 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:21:14.366842 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.367562 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.369656 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.370142 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.370306 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:14.373736 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:21:14.373742 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:21:14.373752 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:21:14.373783 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:21:14.376648 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:14.377553 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:14.377558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:14.377560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:14.377705 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:14.378134 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:14.378138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:14.378140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:14.378555 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:21:14.378682 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:14.379360 (+ 678us) server_negotiation.cc:299] Negotiation successful
0928 02:21:14.379612 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 02:21:14.380754 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:14.362371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.195:57785)
0928 02:21:14.363151 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:14.363194 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:21:14.364983 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:21:14.365669 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:21:14.365676 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:21:14.366155 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:21:14.366697 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:14.366714 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.367726 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.367732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:14.369517 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:14.369532 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.370384 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.370395 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:14.370650 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:14.371357 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:21:14.371379 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:21:14.373524 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:21:14.376798 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:21:14.376807 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:21:14.376822 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:21:14.377298 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:21:14.377815 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:21:14.377818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:21:14.377821 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:21:14.377996 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:21:14.378675 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:21:14.378681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:21:14.378989 (+ 308us) client_negotiation.cc:769] Sending connection context
0928 02:21:14.379218 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 02:21:14.379600 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":351,"threads_started":1}
I20260928 02:21:14.382268 20621 heartbeater.cc:347] Connected to a master server at 127.4.140.254:32967
I20260928 02:21:14.382581 20621 heartbeater.cc:464] Registering TS with master...
I20260928 02:21:14.383397 20621 heartbeater.cc:511] Master 127.4.140.254:32967 requested a full tablet report, sending...
I20260928 02:21:14.385066 20165 ts_manager.cc:194] Registered new tserver with Master: 3e51bb23a4dc4b89ae61691398347cc5 (127.4.140.195:44857)
I20260928 02:21:14.385890 20165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:57785
I20260928 02:21:14.396478 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:21:14.414023 20356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:14.401437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44832 (local address 127.4.140.254:32967)
0928 02:21:14.401649 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:14.401654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:14.402117 (+ 463us) server_negotiation.cc:407] Connection header received
0928 02:21:14.402255 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:14.402259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:14.402326 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:14.402423 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:21:14.403798 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.404664 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.406008 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.406400 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.406510 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:14.408898 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:21:14.408902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:21:14.408905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:21:14.408943 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:21:14.411577 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:14.412306 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:14.412309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:14.412312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:14.412434 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:14.412830 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:14.412834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:14.412836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:14.413174 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:21:14.413362 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:14.413629 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 02:21:14.413786 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 02:21:14.418114 20165 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:21:14.424954 20165 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:21:14.450320 20633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:14.437882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:44857 (local address 127.0.0.1:58030)
0928 02:21:14.440721 (+ 2839us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:14.440740 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:21:14.440864 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:21:14.443167 (+ 2303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:21:14.443175 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:21:14.443209 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:21:14.443609 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:14.443620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.445659 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.445664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:14.447313 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:14.447331 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.449337 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.449343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:14.449515 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:14.449548 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:21:14.449609 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 02:21:14.449878 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2713,"mutex_wait_us":57,"thread_start_us":1126,"threads_started":1}
I20260928 02:21:14.450397 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:14.437691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.4.140.194:37981)
0928 02:21:14.440423 (+ 2732us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:14.440435 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:14.440480 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:21:14.440594 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:14.440604 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:14.441231 (+ 627us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:14.441440 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:21:14.442524 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.444186 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.447017 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.448618 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.448769 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:14.449007 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:14.449825 (+ 818us) server_negotiation.cc:299] Negotiation successful
0928 02:21:14.450018 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2556,"thread_start_us":2411,"threads_started":1}
I20260928 02:21:14.450630 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:14.438691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58030 (local address 127.4.140.195:44857)
0928 02:21:14.442463 (+ 3772us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:14.442473 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:14.442509 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:21:14.442609 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:14.442621 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:14.442937 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:14.443158 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:21:14.443835 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.445487 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.447540 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.449087 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.449762 (+ 675us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:14.450010 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:14.450077 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 02:21:14.450248 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3573,"thread_start_us":3375,"threads_started":1}
I20260928 02:21:14.451154 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:14.437410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:37981 (local address 127.0.0.1:39656)
0928 02:21:14.439411 (+ 2001us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:14.439476 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 02:21:14.439723 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:21:14.441775 (+ 2052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:21:14.441783 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:21:14.441819 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:21:14.442265 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:14.442281 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.444337 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.444342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:14.446818 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:14.446835 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.449152 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.449163 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:14.449400 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:14.449469 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 02:21:14.449609 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 02:21:14.450702 (+ 1093us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1630,"mutex_wait_us":842,"thread_start_us":3040,"threads_started":1}
I20260928 02:21:14.452051 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:14.441509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:39591 (local address 127.0.0.1:58124)
0928 02:21:14.442716 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:14.442739 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:21:14.442914 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:21:14.445043 (+ 2129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:21:14.445048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:21:14.445073 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:21:14.445667 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:14.445682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.447423 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.447428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:14.448898 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:14.448914 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.451287 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.451292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:14.451452 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:14.451484 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:21:14.451578 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 02:21:14.451713 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"thread_start_us":918,"threads_started":1}
I20260928 02:21:14.452363 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:14.442215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58124 (local address 127.4.140.193:39591)
0928 02:21:14.443978 (+ 1763us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:14.443987 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:14.444015 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:21:14.444110 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:14.444120 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:14.444579 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:14.444771 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:21:14.445862 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.447284 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.449074 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.451060 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.451226 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:14.451491 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:14.451788 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 02:21:14.452009 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1510,"thread_start_us":1346,"threads_started":1}
I20260928 02:21:14.455120 20423 tablet_service.cc:1467] Processing CreateTablet for tablet 70d5b3895c294d11a8d28cc4bf34f025 (DEFAULT_TABLE table=test-table [id=6ea40cbeba734d1ba9e92e688365cfd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:21:14.455188 20556 tablet_service.cc:1467] Processing CreateTablet for tablet 70d5b3895c294d11a8d28cc4bf34f025 (DEFAULT_TABLE table=test-table [id=6ea40cbeba734d1ba9e92e688365cfd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:21:14.455752 20289 tablet_service.cc:1467] Processing CreateTablet for tablet 70d5b3895c294d11a8d28cc4bf34f025 (DEFAULT_TABLE table=test-table [id=6ea40cbeba734d1ba9e92e688365cfd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:21:14.456475 20423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70d5b3895c294d11a8d28cc4bf34f025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:21:14.456753 20556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70d5b3895c294d11a8d28cc4bf34f025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:21:14.457147 20289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70d5b3895c294d11a8d28cc4bf34f025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:21:14.468281 20639 tablet_bootstrap.cc:492] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031: Bootstrap starting.
I20260928 02:21:14.471349 20639 tablet_bootstrap.cc:654] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031: Neither blocks nor log segments found. Creating new log.
I20260928 02:21:14.471817 20640 tablet_bootstrap.cc:492] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8: Bootstrap starting.
I20260928 02:21:14.472756 20639 log.cc:824] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031: Log is configured to *not* fsync() on all Append() calls
I20260928 02:21:14.473670 20641 tablet_bootstrap.cc:492] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5: Bootstrap starting.
I20260928 02:21:14.475710 20640 tablet_bootstrap.cc:654] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8: Neither blocks nor log segments found. Creating new log.
I20260928 02:21:14.476998 20640 log.cc:824] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:21:14.477449 20639 tablet_bootstrap.cc:492] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031: No bootstrap required, opened a new log
I20260928 02:21:14.477681 20641 tablet_bootstrap.cc:654] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5: Neither blocks nor log segments found. Creating new log.
I20260928 02:21:14.477788 20639 ts_tablet_manager.cc:1397] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 02:21:14.478927 20641 log.cc:824] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:21:14.480223 20640 tablet_bootstrap.cc:492] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8: No bootstrap required, opened a new log
I20260928 02:21:14.480504 20640 ts_tablet_manager.cc:1397] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260928 02:21:14.482053 20641 tablet_bootstrap.cc:492] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5: No bootstrap required, opened a new log
I20260928 02:21:14.482307 20641 ts_tablet_manager.cc:1397] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 02:21:14.488159 20640 raft_consensus.cc:348] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 } } peers { permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 } } peers { permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 } }
I20260928 02:21:14.488523 20640 raft_consensus.cc:374] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:21:14.488619 20640 raft_consensus.cc:729] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd09ff4c20d44abab01c6570962c8df8, State: Initialized, Role: FOLLOWER
I20260928 02:21:14.488392 20639 raft_consensus.cc:348] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 } } peers { permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 } } peers { permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 } }
I20260928 02:21:14.488778 20639 raft_consensus.cc:374] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:21:14.488874 20639 raft_consensus.cc:729] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bb12dd2a4d14e1f9d9470a75e58e031, State: Initialized, Role: FOLLOWER
I20260928 02:21:14.489318 20640 consensus_queue.cc:260] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 } } peers { permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 } } peers { permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 } }
I20260928 02:21:14.489493 20639 consensus_queue.cc:260] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 } } peers { permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 } } peers { permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 } }
I20260928 02:21:14.491077 20640 ts_tablet_manager.cc:1428] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260928 02:21:14.492017 20488 heartbeater.cc:503] Master 127.4.140.254:32967 was elected leader, sending a full tablet report...
I20260928 02:21:14.492669 20639 ts_tablet_manager.cc:1428] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260928 02:21:14.493052 20641 raft_consensus.cc:348] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 } } peers { permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 } } peers { permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 } }
I20260928 02:21:14.493498 20641 raft_consensus.cc:374] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:21:14.493604 20641 raft_consensus.cc:729] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e51bb23a4dc4b89ae61691398347cc5, State: Initialized, Role: FOLLOWER
I20260928 02:21:14.494194 20641 consensus_queue.cc:260] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 } } peers { permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 } } peers { permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 } }
I20260928 02:21:14.495517 20621 heartbeater.cc:503] Master 127.4.140.254:32967 was elected leader, sending a full tablet report...
I20260928 02:21:14.495860 20641 ts_tablet_manager.cc:1428] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260928 02:21:14.578790 20355 tablet.cc:2369] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:21:14.593374 20489 tablet.cc:2369] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:21:14.610911 20622 tablet.cc:2369] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:21:14.618342 20647 raft_consensus.cc:482] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:21:14.618660 20647 raft_consensus.cc:504] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 } } peers { permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 } } peers { permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 } }
I20260928 02:21:14.620407 20647 leader_election.cc:288] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bb12dd2a4d14e1f9d9470a75e58e031 (127.4.140.194:37981), dd09ff4c20d44abab01c6570962c8df8 (127.4.140.193:39591)
I20260928 02:21:14.627516 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:14.620778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:37981 (local address 127.4.140.195:57019)
0928 02:21:14.621144 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:14.621163 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:21:14.621298 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:21:14.621670 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:21:14.621674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:21:14.621706 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:21:14.621888 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:14.621896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.623313 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.623319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:14.625105 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:14.625118 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.626681 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.626687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:14.626883 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:14.626943 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 02:21:14.627065 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 02:21:14.627215 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":1}
I20260928 02:21:14.627562 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:14.620945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:57019 (local address 127.4.140.194:37981)
0928 02:21:14.621206 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:14.621214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:14.621239 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:21:14.621475 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:14.621485 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:14.621546 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:14.621647 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:21:14.622068 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.623152 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.625269 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.626535 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.626681 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:14.626784 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:14.627216 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 02:21:14.627339 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260928 02:21:14.628234 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:14.621030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:39591 (local address 127.4.140.195:60195)
0928 02:21:14.621847 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:14.621860 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:21:14.621960 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:21:14.622350 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:21:14.622355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:21:14.622393 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:21:14.622773 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:14.622785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.624286 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.624290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:14.626021 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:14.626037 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.627521 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.627528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:14.627687 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:14.627718 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:21:14.627805 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:21:14.627963 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":483,"threads_started":1}
I20260928 02:21:14.628279 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:14.620981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:60195 (local address 127.4.140.193:39591)
0928 02:21:14.621222 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:14.621227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:14.621951 (+ 724us) server_negotiation.cc:407] Connection header received
0928 02:21:14.622140 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:14.622144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:14.622204 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:14.622282 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:21:14.622970 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.624138 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.626170 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.627320 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.627489 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:14.627560 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:14.627873 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 02:21:14.628032 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:21:14.628872 20443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d5b3895c294d11a8d28cc4bf34f025" candidate_uuid: "3e51bb23a4dc4b89ae61691398347cc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" is_pre_election: true
I20260928 02:21:14.629416 20443 raft_consensus.cc:2454] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e51bb23a4dc4b89ae61691398347cc5 in term 0.
I20260928 02:21:14.629367 20309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d5b3895c294d11a8d28cc4bf34f025" candidate_uuid: "3e51bb23a4dc4b89ae61691398347cc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd09ff4c20d44abab01c6570962c8df8" is_pre_election: true
I20260928 02:21:14.629918 20309 raft_consensus.cc:2454] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e51bb23a4dc4b89ae61691398347cc5 in term 0.
I20260928 02:21:14.630115 20512 leader_election.cc:302] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e51bb23a4dc4b89ae61691398347cc5, 7bb12dd2a4d14e1f9d9470a75e58e031; no voters:
I20260928 02:21:14.630584 20647 raft_consensus.cc:2781] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:21:14.630740 20647 raft_consensus.cc:482] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:21:14.630846 20647 raft_consensus.cc:3037] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:21:14.633060 20647 raft_consensus.cc:504] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 } } peers { permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 } } peers { permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 } }
I20260928 02:21:14.633637 20647 leader_election.cc:288] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [CANDIDATE]: Term 1 election: Requested vote from peers 7bb12dd2a4d14e1f9d9470a75e58e031 (127.4.140.194:37981), dd09ff4c20d44abab01c6570962c8df8 (127.4.140.193:39591)
I20260928 02:21:14.634087 20309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d5b3895c294d11a8d28cc4bf34f025" candidate_uuid: "3e51bb23a4dc4b89ae61691398347cc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd09ff4c20d44abab01c6570962c8df8"
I20260928 02:21:14.634109 20443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d5b3895c294d11a8d28cc4bf34f025" candidate_uuid: "3e51bb23a4dc4b89ae61691398347cc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031"
I20260928 02:21:14.634292 20443 raft_consensus.cc:3037] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:21:14.634294 20309 raft_consensus.cc:3037] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:21:14.637519 20309 raft_consensus.cc:2454] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e51bb23a4dc4b89ae61691398347cc5 in term 1.
I20260928 02:21:14.637533 20443 raft_consensus.cc:2454] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e51bb23a4dc4b89ae61691398347cc5 in term 1.
I20260928 02:21:14.637923 20512 leader_election.cc:302] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e51bb23a4dc4b89ae61691398347cc5, 7bb12dd2a4d14e1f9d9470a75e58e031; no voters:
I20260928 02:21:14.638203 20647 raft_consensus.cc:2781] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:21:14.638628 20647 raft_consensus.cc:686] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [term 1 LEADER]: Becoming Leader. State: Replica: 3e51bb23a4dc4b89ae61691398347cc5, State: Running, Role: LEADER
I20260928 02:21:14.639181 20647 consensus_queue.cc:237] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 } } peers { permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 } } peers { permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 } }
I20260928 02:21:14.643994 20164 catalog_manager.cc:5495] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e51bb23a4dc4b89ae61691398347cc5 (127.4.140.195). New cstate: current_term: 1 leader_uuid: "3e51bb23a4dc4b89ae61691398347cc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 } health_report { overall_health: UNKNOWN } } }
I20260928 02:21:14.686940 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:14.680906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58036 (local address 127.4.140.195:44857)
0928 02:21:14.681127 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:14.681134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:14.681246 (+ 112us) server_negotiation.cc:407] Connection header received
0928 02:21:14.681501 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:14.681505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:14.681591 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:14.681758 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:21:14.682166 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.682906 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.684494 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.685122 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.685254 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:14.685718 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:21:14.686400 (+ 682us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:21:14.686544 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:14.686596 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 02:21:14.686680 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:21:14.695387 20443 raft_consensus.cc:1261] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 1 FOLLOWER]: Refusing update from remote peer 3e51bb23a4dc4b89ae61691398347cc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:21:14.695386 20309 raft_consensus.cc:1261] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8 [term 1 FOLLOWER]: Refusing update from remote peer 3e51bb23a4dc4b89ae61691398347cc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:21:14.696451 20649 consensus_queue.cc:1035] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:21:14.696843 20647 consensus_queue.cc:1035] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:21:14.707319 20652 mvcc.cc:205] Tried to move back new op lower bound from 7334142257941389312 to 7334142257726263296. Current Snapshot: MvccSnapshot[applied={T|T < 7334142257941389312}]
I20260928 02:21:14.709252 20654 mvcc.cc:205] Tried to move back new op lower bound from 7334142257941389312 to 7334142257726263296. Current Snapshot: MvccSnapshot[applied={T|T < 7334142257941389312}]
I20260928 02:21:14.724584 20165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:21:14.724877 20165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:21:14.728734 20165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb: Sending DeleteTablet for 3 replicas of tablet 70d5b3895c294d11a8d28cc4bf34f025
I20260928 02:21:14.729959 20423 tablet_service.cc:1514] Processing DeleteTablet for tablet 70d5b3895c294d11a8d28cc4bf34f025 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:21:14 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:39656
I20260928 02:21:14.730151 20556 tablet_service.cc:1514] Processing DeleteTablet for tablet 70d5b3895c294d11a8d28cc4bf34f025 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:21:14 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:58030
I20260928 02:21:14.730891 20289 tablet_service.cc:1514] Processing DeleteTablet for tablet 70d5b3895c294d11a8d28cc4bf34f025 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:21:14 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:58124
I20260928 02:21:14.731158 20662 tablet_replica.cc:330] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5: stopping tablet replica
I20260928 02:21:14.731580 20662 raft_consensus.cc:2229] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:21:14.731873 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 20226
I20260928 02:21:14.732332 20662 raft_consensus.cc:2258] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:21:14.737633 20662 ts_tablet_manager.cc:1905] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:21:14.736542 20663 tablet_replica.cc:330] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8: stopping tablet replica
I20260928 02:21:14.738703 20663 raft_consensus.cc:2229] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:21:14.739404 20663 raft_consensus.cc:2258] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 28 02:21:14 dist-test-slave-h37r krb5kdc[20120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:21:14.745834 20662 ts_tablet_manager.cc:1918] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:21:14.745378 20663 ts_tablet_manager.cc:1905] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:21:14.746032 20662 log.cc:1196] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/70d5b3895c294d11a8d28cc4bf34f025
I20260928 02:21:14.746577 20662 ts_tablet_manager.cc:1939] T 70d5b3895c294d11a8d28cc4bf34f025 P 3e51bb23a4dc4b89ae61691398347cc5: Deleting consensus metadata
I20260928 02:21:14.749054 20151 catalog_manager.cc:4841] TS 3e51bb23a4dc4b89ae61691398347cc5 (127.4.140.195:44857): tablet 70d5b3895c294d11a8d28cc4bf34f025 (table test-table [id=6ea40cbeba734d1ba9e92e688365cfd7]) successfully deleted
I20260928 02:21:14.749974 20661 tablet_replica.cc:330] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031: stopping tablet replica
I20260928 02:21:14.750435 20661 raft_consensus.cc:2229] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:21:14.751045 20661 raft_consensus.cc:2258] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:21:14.753055 20661 ts_tablet_manager.cc:1905] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:21:14.755436 20663 ts_tablet_manager.cc:1918] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:21:14.755580 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:14.733404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58126 (local address 127.4.140.193:39591)
0928 02:21:14.733583 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:14.733594 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:14.733608 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:21:14.733650 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:14.733653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:14.733718 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:14.733842 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:21:14.736317 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.737910 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.739844 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.740510 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.740609 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:14.746622 (+ 6013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:21:14.746639 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:21:14.746656 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:21:14.746703 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:21:14.751028 (+ 4325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:14.752435 (+ 1407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:14.752443 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:14.752446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:14.752747 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:14.753277 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:14.753282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:14.753285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:14.753795 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:21:14.754083 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:14.754796 (+ 713us) server_negotiation.cc:299] Negotiation successful
0928 02:21:14.755051 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:21:14.755925 20663 log.cc:1196] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/70d5b3895c294d11a8d28cc4bf34f025
I20260928 02:21:14.756732 20663 ts_tablet_manager.cc:1939] T 70d5b3895c294d11a8d28cc4bf34f025 P dd09ff4c20d44abab01c6570962c8df8: Deleting consensus metadata
I20260928 02:21:14.757236 20349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:21:14.764835 20661 ts_tablet_manager.cc:1918] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:21:14.765055 20661 log.cc:1196] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/70d5b3895c294d11a8d28cc4bf34f025
I20260928 02:21:14.765619 20661 ts_tablet_manager.cc:1939] T 70d5b3895c294d11a8d28cc4bf34f025 P 7bb12dd2a4d14e1f9d9470a75e58e031: Deleting consensus metadata
I20260928 02:21:14.767663 20152 catalog_manager.cc:4841] TS 7bb12dd2a4d14e1f9d9470a75e58e031 (127.4.140.194:37981): tablet 70d5b3895c294d11a8d28cc4bf34f025 (table test-table [id=6ea40cbeba734d1ba9e92e688365cfd7]) successfully deleted
I20260928 02:21:14.856945 20150 catalog_manager.cc:4841] TS dd09ff4c20d44abab01c6570962c8df8 (127.4.140.193:39591): tablet 70d5b3895c294d11a8d28cc4bf34f025 (table test-table [id=6ea40cbeba734d1ba9e92e688365cfd7]) successfully deleted
I20260928 02:21:14.857604 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 20226
I20260928 02:21:14.877617 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 20360
Sep 28 02:21:14 dist-test-slave-h37r krb5kdc[20120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.194@KRBTEST.COM
I20260928 02:21:14.895486 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:14.878650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39658 (local address 127.4.140.194:37981)
0928 02:21:14.878847 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:14.878853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:14.878878 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:21:14.878969 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:14.878975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:14.879072 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:14.879193 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:21:14.880697 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.881858 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.883211 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:14.883849 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:14.884016 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:14.888494 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:21:14.888515 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:21:14.888536 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:21:14.888583 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:21:14.892458 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:14.893202 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:14.893211 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:14.893215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:14.893369 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:14.893742 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:14.893748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:14.893751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:14.894269 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:21:14.894585 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:14.894831 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 02:21:14.895075 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:21:14.896502 20483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:21:14.988721 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 20360
I20260928 02:21:15.007220 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 20493
Sep 28 02:21:15 dist-test-slave-h37r krb5kdc[20120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.195@KRBTEST.COM
I20260928 02:21:15.026001 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:15.008115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58042 (local address 127.4.140.195:44857)
0928 02:21:15.008322 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:15.008330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:15.008354 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:21:15.008509 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:15.008515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:15.008617 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:15.008741 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:21:15.010084 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:15.011332 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:15.013061 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:15.013747 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:15.013888 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:15.018650 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:21:15.018666 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:21:15.018692 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:21:15.018744 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:21:15.022821 (+ 4077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:15.023566 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:15.023575 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:15.023578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:15.023783 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:15.024251 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:15.024256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:15.024259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:15.024713 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:21:15.025033 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:15.025270 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 02:21:15.025508 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:21:15.026750 20616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:21:15.151054 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 20493
I20260928 02:21:15.170573 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 20133
I20260928 02:21:15.171742 20195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:21:15.284276 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 20133
I20260928 02:21:35.307305 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:32967
--webserver_interface=127.4.140.254
--webserver_port=40115
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:32967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:21:35.639321 20669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:21:35.639621 20669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:21:35.639775 20669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:21:35.647069 20669 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:21:35.647168 20669 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:21:35.647229 20669 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:21:35.647308 20669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:21:35.647403 20669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:21:35.647516 20669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:21:35.647591 20669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:21:35.647684 20669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:21:35.657949 20669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:32967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:32967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=40115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:21:35.659238 20669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:21:35.660614 20669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:21:35.683462 20676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:35.683439 20678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:35.683561 20675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:21:35.684046 20669 server_base.cc:1034] running on GCE node
I20260928 02:21:35.684963 20669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:21:35.686527 20669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:21:35.687783 20669 hybrid_clock.cc:648] HybridClock initialized: now 1790562095687704 us; error 54 us; skew 500 ppm
I20260928 02:21:35.688213 20669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:21:35 dist-test-slave-h37r krb5kdc[20120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562095, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:21:35.691838 20669 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:21:35.693408 20669 webserver.cc:460] Webserver started at http://127.4.140.254:40115/ using document root <none> and password file <none>
I20260928 02:21:35.694139 20669 fs_manager.cc:359] Metadata directory not provided
I20260928 02:21:35.694243 20669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:21:35.700443 20669 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260928 02:21:35.703358 20685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:35.704825 20669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:21:35.705000 20669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "6f3cfae4688a4df3bce475f9158ac2bb"
format_stamp: "Formatted at 2026-09-28 02:21:12 on dist-test-slave-h37r"
server_key: "2191e889c53695688ed10ac79da3fc5c"
server_key_iv: "2e0ad528d61e3705115c7c170c3ced9e"
server_key_version: "encryptionkey@0"
I20260928 02:21:35.706012 20669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:21:35.719867 20669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:21:35.724061 20669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:21:35.724305 20669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:21:35.743047 20669 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:32967
I20260928 02:21:35.743089 20736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:32967 every 8 connection(s)
I20260928 02:21:35.744709 20669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:21:35.751837 20737 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:21:35.752307 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 20669
I20260928 02:21:35.753331 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:39591
--local_ip_for_outbound_sockets=127.4.140.193
--tserver_master_addrs=127.4.140.254:32967
--webserver_port=42207
--webserver_interface=127.4.140.193
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:21:35.757015 20737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb: Bootstrap starting.
I20260928 02:21:35.778318 20737 log.cc:824] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:21:35.790304 20737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:21:35.790841 20737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb: Bootstrap complete.
I20260928 02:21:35.801074 20737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32967 } }
I20260928 02:21:35.801960 20737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3cfae4688a4df3bce475f9158ac2bb, State: Initialized, Role: FOLLOWER
I20260928 02:21:35.802620 20737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32967 } }
I20260928 02:21:35.802874 20737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:21:35.803006 20737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:21:35.803177 20737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:21:35.807034 20737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32967 } }
I20260928 02:21:35.807587 20737 leader_election.cc:302] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f3cfae4688a4df3bce475f9158ac2bb; no voters:
I20260928 02:21:35.808400 20737 leader_election.cc:288] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:21:35.808594 20741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:21:35.809832 20741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 2 LEADER]: Becoming Leader. State: Replica: 6f3cfae4688a4df3bce475f9158ac2bb, State: Running, Role: LEADER
I20260928 02:21:35.810436 20741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32967 } }
I20260928 02:21:35.810972 20737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:21:35.816851 20742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32967 } } }
I20260928 02:21:35.816886 20743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f3cfae4688a4df3bce475f9158ac2bb. Latest consensus state: current_term: 2 leader_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32967 } } }
I20260928 02:21:35.817368 20743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [sys.catalog]: This master's current role is: LEADER
I20260928 02:21:35.817260 20742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [sys.catalog]: This master's current role is: LEADER
I20260928 02:21:35.823570 20750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:21:35.831746 20750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:21:35.833133 20750 catalog_manager.cc:1259] Loaded cluster ID: 1720ff3f24594a0f95df1c88a5911f1d
I20260928 02:21:35.833225 20750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:21:35.837294 20750 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:21:35.838780 20750 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb: Loaded TSK: 0
I20260928 02:21:35.872202 20750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb: Generated new TSK 1
I20260928 02:21:35.872608 20750 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:21:36.105754 20739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:21:36.106024 20739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:21:36.106168 20739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:21:36.113978 20739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:21:36.114094 20739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:21:36.114267 20739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:21:36.124454 20739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:39591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=42207
--tserver_master_addrs=127.4.140.254:32967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:21:36.125803 20739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:21:36.127137 20739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:21:36.140293 20764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:36.140337 20765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:36.140934 20767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:21:36.141475 20739 server_base.cc:1034] running on GCE node
I20260928 02:21:36.142150 20739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:21:36.143325 20739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:21:36.144579 20739 hybrid_clock.cc:648] HybridClock initialized: now 1790562096144502 us; error 60 us; skew 500 ppm
I20260928 02:21:36.144970 20739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:21:36 dist-test-slave-h37r krb5kdc[20120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562096, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:21:36.149137 20739 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:21:36.150941 20739 webserver.cc:460] Webserver started at http://127.4.140.193:42207/ using document root <none> and password file <none>
I20260928 02:21:36.151861 20739 fs_manager.cc:359] Metadata directory not provided
I20260928 02:21:36.152050 20739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:21:36.159318 20739 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:21:36.162748 20774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:36.164415 20739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 02:21:36.164593 20739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "dd09ff4c20d44abab01c6570962c8df8"
format_stamp: "Formatted at 2026-09-28 02:21:13 on dist-test-slave-h37r"
server_key: "1362e6cc326596ae57af4dbad5196012"
server_key_iv: "77206a69e16a1db68e35e9cbafd99098"
server_key_version: "encryptionkey@0"
I20260928 02:21:36.165946 20739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:21:36.184037 20739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:21:36.189538 20739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:21:36.189859 20739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:21:36.191066 20739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:21:36.192813 20739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:21:36.192921 20739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:36.193017 20739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:21:36.193091 20739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:36.237938 20739 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:39591
I20260928 02:21:36.237973 20886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:39591 every 8 connection(s)
I20260928 02:21:36.239430 20739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:21:36.240702 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 20739
I20260928 02:21:36.241509 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:37981
--local_ip_for_outbound_sockets=127.4.140.194
--tserver_master_addrs=127.4.140.254:32967
--webserver_port=37725
--webserver_interface=127.4.140.194
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:21:36 dist-test-slave-h37r krb5kdc[20120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562096, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:21:36.263361 20889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:36.242925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:48881 (local address 127.4.140.254:32967)
0928 02:21:36.243563 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:36.243580 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:36.244186 (+ 606us) server_negotiation.cc:407] Connection header received
0928 02:21:36.246376 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:36.246395 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:36.246851 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:36.247263 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:21:36.248702 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:36.249935 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:36.251461 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:36.252193 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:36.252388 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:36.255802 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:21:36.255809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:21:36.255823 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:21:36.255863 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:21:36.258814 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:36.259560 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:36.259567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:36.259573 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:36.259748 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:36.260201 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:36.260206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:36.260209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:36.260690 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:21:36.260884 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:36.261675 (+ 791us) server_negotiation.cc:299] Negotiation successful
0928 02:21:36.262051 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":249,"threads_started":1}
I20260928 02:21:36.263355 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:36.243069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.193:48881)
0928 02:21:36.243986 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:36.244040 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:21:36.245774 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:21:36.247371 (+ 1597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:21:36.247378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:21:36.247870 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:21:36.248488 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:36.248505 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:36.250096 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:36.250101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:36.251308 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:36.251320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:36.252431 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:36.252439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:36.252708 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:36.253412 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:21:36.253434 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:21:36.255540 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:21:36.258970 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:21:36.258979 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:21:36.258993 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:21:36.259410 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:21:36.259863 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:21:36.259867 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:21:36.259871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:21:36.260082 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:21:36.260826 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:21:36.260833 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:21:36.261167 (+ 334us) client_negotiation.cc:769] Sending connection context
0928 02:21:36.261441 (+ 274us) client_negotiation.cc:240] Negotiation successful
0928 02:21:36.261995 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":400,"threads_started":1}
I20260928 02:21:36.266414 20887 heartbeater.cc:347] Connected to a master server at 127.4.140.254:32967
I20260928 02:21:36.266757 20887 heartbeater.cc:464] Registering TS with master...
I20260928 02:21:36.267596 20887 heartbeater.cc:511] Master 127.4.140.254:32967 requested a full tablet report, sending...
I20260928 02:21:36.269914 20702 ts_manager.cc:194] Registered new tserver with Master: dd09ff4c20d44abab01c6570962c8df8 (127.4.140.193:39591)
I20260928 02:21:36.271543 20702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:48881
W20260928 02:21:36.571662 20891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:21:36.571955 20891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:21:36.572125 20891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:21:36.579299 20891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:21:36.579396 20891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:21:36.579587 20891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:21:36.589231 20891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:37981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=37725
--tserver_master_addrs=127.4.140.254:32967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:21:36.590509 20891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:21:36.591805 20891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:21:36.611577 20896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:36.611635 20899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:36.611673 20897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:21:36.612516 20891 server_base.cc:1034] running on GCE node
I20260928 02:21:36.613138 20891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:21:36.614104 20891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:21:36.615329 20891 hybrid_clock.cc:648] HybridClock initialized: now 1790562096615282 us; error 42 us; skew 500 ppm
I20260928 02:21:36.615651 20891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:21:36 dist-test-slave-h37r krb5kdc[20120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562096, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:21:36.618767 20891 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:21:36.620344 20891 webserver.cc:460] Webserver started at http://127.4.140.194:37725/ using document root <none> and password file <none>
I20260928 02:21:36.621025 20891 fs_manager.cc:359] Metadata directory not provided
I20260928 02:21:36.621183 20891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:21:36.626744 20891 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:21:36.629463 20906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:36.630702 20891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:21:36.630898 20891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "7bb12dd2a4d14e1f9d9470a75e58e031"
format_stamp: "Formatted at 2026-09-28 02:21:13 on dist-test-slave-h37r"
server_key: "7b89d87cd2c3076431081e3ffb1ffc8f"
server_key_iv: "91ed23319b8c1a56f68a792ccfd25f19"
server_key_version: "encryptionkey@0"
I20260928 02:21:36.631851 20891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:21:36.662676 20891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:21:36.667310 20891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:21:36.667603 20891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:21:36.668653 20891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:21:36.670024 20891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:21:36.670142 20891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:36.670284 20891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:21:36.670368 20891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:36.703943 20891 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:37981
I20260928 02:21:36.704015 21018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:37981 every 8 connection(s)
I20260928 02:21:36.705375 20891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:21:36.706830 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 20891
I20260928 02:21:36.707454 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:44857
--local_ip_for_outbound_sockets=127.4.140.195
--tserver_master_addrs=127.4.140.254:32967
--webserver_port=43961
--webserver_interface=127.4.140.195
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:21:36 dist-test-slave-h37r krb5kdc[20120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562096, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:21:36.731130 20889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:36.708248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:36369 (local address 127.4.140.254:32967)
0928 02:21:36.708462 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:36.708468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:36.715205 (+ 6737us) server_negotiation.cc:407] Connection header received
0928 02:21:36.717207 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:36.717212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:36.717284 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:36.717417 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:21:36.718976 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:36.719675 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:36.721312 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:36.721745 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:36.721877 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:36.725528 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:21:36.725533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:21:36.725536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:21:36.725568 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:21:36.728130 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:36.728846 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:36.728851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:36.728853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:36.728972 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:36.729401 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:36.729406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:36.729410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:36.729689 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:21:36.729849 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:36.730533 (+ 684us) server_negotiation.cc:299] Negotiation successful
0928 02:21:36.730772 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:21:36.732163 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:36.708477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.194:36369)
0928 02:21:36.714975 (+ 6498us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:36.715031 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:21:36.716957 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:21:36.717679 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:21:36.717688 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:21:36.718211 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:21:36.718798 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:36.718815 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:36.719828 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:36.719838 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:36.721159 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:36.721176 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:36.721996 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:36.722004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:36.722235 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:36.722951 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:21:36.722973 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:21:36.725308 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:21:36.728260 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:21:36.728269 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:21:36.728286 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:21:36.728713 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:21:36.729078 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:21:36.729083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:21:36.729087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:21:36.729280 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:21:36.729816 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:21:36.729825 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:21:36.730178 (+ 353us) client_negotiation.cc:769] Sending connection context
0928 02:21:36.730440 (+ 262us) client_negotiation.cc:240] Negotiation successful
0928 02:21:36.730975 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6119,"thread_start_us":5905,"threads_started":1}
I20260928 02:21:36.733680 21019 heartbeater.cc:347] Connected to a master server at 127.4.140.254:32967
I20260928 02:21:36.733973 21019 heartbeater.cc:464] Registering TS with master...
I20260928 02:21:36.734654 21019 heartbeater.cc:511] Master 127.4.140.254:32967 requested a full tablet report, sending...
I20260928 02:21:36.736234 20702 ts_manager.cc:194] Registered new tserver with Master: 7bb12dd2a4d14e1f9d9470a75e58e031 (127.4.140.194:37981)
I20260928 02:21:36.737334 20702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:36369
W20260928 02:21:37.043239 21022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:21:37.043511 21022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:21:37.043682 21022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:21:37.051218 21022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:21:37.051396 21022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:21:37.051625 21022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:21:37.062817 21022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:44857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=43961
--tserver_master_addrs=127.4.140.254:32967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:21:37.064245 21022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:21:37.065680 21022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:21:37.079804 21028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:37.079850 21027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:37.079804 21030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:21:37.080806 21022 server_base.cc:1034] running on GCE node
I20260928 02:21:37.081455 21022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:21:37.082428 21022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:21:37.083691 21022 hybrid_clock.cc:648] HybridClock initialized: now 1790562097083637 us; error 43 us; skew 500 ppm
I20260928 02:21:37.084117 21022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:21:37 dist-test-slave-h37r krb5kdc[20120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562097, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:21:37.087277 21022 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:21:37.088794 21022 webserver.cc:460] Webserver started at http://127.4.140.195:43961/ using document root <none> and password file <none>
I20260928 02:21:37.089480 21022 fs_manager.cc:359] Metadata directory not provided
I20260928 02:21:37.089639 21022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:21:37.095142 21022 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:21:37.098277 21037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:37.099872 21022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:21:37.100116 21022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "3e51bb23a4dc4b89ae61691398347cc5"
format_stamp: "Formatted at 2026-09-28 02:21:14 on dist-test-slave-h37r"
server_key: "2282be1320dfbfa766f8d74f257ef412"
server_key_iv: "cf8591c557f40ffdca81db044c199cc9"
server_key_version: "encryptionkey@0"
I20260928 02:21:37.101171 21022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:21:37.117202 21022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:21:37.121948 21022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:21:37.122254 21022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:21:37.123402 21022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:21:37.125048 21022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:21:37.125180 21022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:37.125317 21022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:21:37.125412 21022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:37.160888 21022 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:44857
I20260928 02:21:37.160944 21149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:44857 every 8 connection(s)
I20260928 02:21:37.162411 21022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:21:37.163630 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 21022
Sep 28 02:21:37 dist-test-slave-h37r krb5kdc[20120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562097, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:21:37.182662 20889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:37.164688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53326 (local address 127.4.140.254:32967)
0928 02:21:37.164874 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:37.164878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:37.165885 (+ 1007us) server_negotiation.cc:407] Connection header received
0928 02:21:37.166096 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:37.166111 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:37.166175 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:37.166304 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:21:37.168366 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.169489 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.171091 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.172091 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.172234 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:37.176662 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:21:37.176666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:21:37.176670 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:21:37.176699 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:21:37.179508 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:37.180281 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:37.180289 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:37.180293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:37.180451 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:37.180965 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:37.180974 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:37.180982 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:37.181298 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:21:37.181693 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:37.181979 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 02:21:37.182290 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":64}
I20260928 02:21:37.185806 21153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:37.165126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:35227 (local address 127.4.140.254:32967)
0928 02:21:37.165568 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:37.165576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:37.166810 (+ 1234us) server_negotiation.cc:407] Connection header received
0928 02:21:37.168828 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:37.168833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:37.168891 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:37.168975 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:21:37.170643 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.171715 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.174243 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.174966 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.175097 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:37.179499 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:21:37.179505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:21:37.179510 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:21:37.179545 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:21:37.182327 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:37.183189 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:37.183194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:37.183198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:37.183316 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:37.183814 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:37.183820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:37.183824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:37.184238 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:21:37.184412 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:37.185153 (+ 741us) server_negotiation.cc:299] Negotiation successful
0928 02:21:37.185373 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":331,"thread_start_us":243,"threads_started":1}
I20260928 02:21:37.186841 21154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:37.165940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.195:35227)
0928 02:21:37.166683 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:37.166717 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:21:37.168535 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:21:37.169226 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:21:37.169237 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:21:37.169852 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:21:37.170479 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:37.170497 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.171936 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.171942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:37.174067 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:37.174101 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.175242 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.175253 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:37.175641 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:37.176405 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:21:37.176438 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:21:37.179254 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:21:37.182521 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:21:37.182532 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:21:37.182549 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:21:37.183057 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:21:37.183456 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:21:37.183462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:21:37.183466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:21:37.183675 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:21:37.184389 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:21:37.184398 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:21:37.184751 (+ 353us) client_negotiation.cc:769] Sending connection context
0928 02:21:37.185071 (+ 320us) client_negotiation.cc:240] Negotiation successful
0928 02:21:37.185638 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":295,"threads_started":1}
I20260928 02:21:37.188215 21150 heartbeater.cc:347] Connected to a master server at 127.4.140.254:32967
I20260928 02:21:37.188521 21150 heartbeater.cc:464] Registering TS with master...
I20260928 02:21:37.189218 21150 heartbeater.cc:511] Master 127.4.140.254:32967 requested a full tablet report, sending...
I20260928 02:21:37.190528 20702 ts_manager.cc:194] Registered new tserver with Master: 3e51bb23a4dc4b89ae61691398347cc5 (127.4.140.195:44857)
I20260928 02:21:37.191339 20702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:35227
I20260928 02:21:37.197355 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:21:37.203948 21153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:37.198308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.4.140.254:32967)
0928 02:21:37.198466 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:37.198470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:37.199078 (+ 608us) server_negotiation.cc:407] Connection header received
0928 02:21:37.199279 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:37.199283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:37.199453 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:37.199568 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:21:37.199999 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.200702 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.202291 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.202666 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.202762 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:37.203267 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:21:37.203472 (+ 205us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 02:21:37.203718 (+ 246us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53340: token expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 02:21:37.204049 21153 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53340: token expired
W20260928 02:21:37.204730 20624 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.140.254:32967)
I20260928 02:21:37.217972 21153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:37.205227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.4.140.254:32967)
0928 02:21:37.205469 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:37.205493 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:37.205583 (+ 90us) server_negotiation.cc:407] Connection header received
0928 02:21:37.205773 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:37.205784 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:37.205836 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:37.205972 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:21:37.207326 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.208090 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.209431 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.209885 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.210026 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:37.212923 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:21:37.212927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:21:37.212930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:21:37.212960 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:21:37.215262 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:37.216004 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:37.216012 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:37.216015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:37.216142 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:37.216549 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:37.216554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:37.216558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:37.216865 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:21:37.217141 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:37.217427 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 02:21:37.217649 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260928 02:21:37.221316 20702 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:21:37.228763 20702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:21:37.252502 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:37.240931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:37981 (local address 127.0.0.1:34570)
0928 02:21:37.242499 (+ 1568us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:37.242557 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:21:37.242765 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:21:37.244741 (+ 1976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:21:37.244752 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:21:37.244788 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:21:37.245273 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:37.245288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.247043 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.247055 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:37.249543 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:37.249560 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.251431 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.251437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:37.251632 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:37.251693 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 02:21:37.251821 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 02:21:37.252049 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1292,"mutex_wait_us":2,"thread_start_us":5111,"threads_started":1}
I20260928 02:21:37.253206 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:37.241152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.4.140.194:37981)
0928 02:21:37.243664 (+ 2512us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:37.243676 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:37.243713 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:21:37.243829 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:37.243840 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:37.244176 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:37.244406 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:21:37.245490 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.246891 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.249734 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.251218 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.251365 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:37.251571 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:37.252750 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0928 02:21:37.252953 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2346,"thread_start_us":2250,"threads_started":1}
I20260928 02:21:37.253901 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:37.242154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47816 (local address 127.4.140.193:39591)
0928 02:21:37.244566 (+ 2412us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:37.244577 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:37.245503 (+ 926us) server_negotiation.cc:407] Connection header received
0928 02:21:37.245743 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:37.245751 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:37.245979 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:37.246154 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:21:37.247107 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.248735 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.250824 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.252162 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.252295 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:37.252507 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:37.253557 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0928 02:21:37.253688 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2232,"thread_start_us":2110,"threads_started":1}
I20260928 02:21:37.254000 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:37.241468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:39591 (local address 127.0.0.1:47816)
0928 02:21:37.245394 (+ 3926us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:37.245413 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:21:37.245541 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:21:37.246390 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:21:37.246396 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:21:37.246420 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:21:37.246877 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:37.246894 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.248917 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.248922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:37.250668 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:37.250680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.252941 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.252947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:37.253151 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:37.253213 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 02:21:37.253508 (+ 295us) client_negotiation.cc:240] Negotiation successful
0928 02:21:37.253632 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3790,"mutex_wait_us":4,"thread_start_us":3642,"threads_started":1}
I20260928 02:21:37.254952 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:37.241772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:44857 (local address 127.0.0.1:32864)
0928 02:21:37.245887 (+ 4115us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:37.245903 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:21:37.246025 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:21:37.246676 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:21:37.246681 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:21:37.246706 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:21:37.247155 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:37.247166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.248812 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.248817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:37.251020 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:37.251034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.253149 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.253154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:37.254351 (+ 1197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:37.254374 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:21:37.254478 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:21:37.254609 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4007,"mutex_wait_us":90,"thread_start_us":737,"threads_started":1}
I20260928 02:21:37.254984 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:37.242153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.4.140.195:44857)
0928 02:21:37.245109 (+ 2956us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:37.245117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:37.246142 (+ 1025us) server_negotiation.cc:407] Connection header received
0928 02:21:37.246242 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:37.246250 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:37.246474 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:37.246664 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:21:37.247335 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.248651 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.251192 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.252684 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.252790 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:37.253004 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:37.254520 (+ 1516us) server_negotiation.cc:299] Negotiation successful
0928 02:21:37.254680 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2767,"thread_start_us":2602,"threads_started":1}
I20260928 02:21:37.257743 20822 tablet_service.cc:1467] Processing CreateTablet for tablet 84ab87b8f82f415d80434ce4ad3e322c (DEFAULT_TABLE table=test-table [id=ae79bd05aac24a629500a446cf9beba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:21:37.257743 20954 tablet_service.cc:1467] Processing CreateTablet for tablet 84ab87b8f82f415d80434ce4ad3e322c (DEFAULT_TABLE table=test-table [id=ae79bd05aac24a629500a446cf9beba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:21:37.258168 21085 tablet_service.cc:1467] Processing CreateTablet for tablet 84ab87b8f82f415d80434ce4ad3e322c (DEFAULT_TABLE table=test-table [id=ae79bd05aac24a629500a446cf9beba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:21:37.259255 20954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ab87b8f82f415d80434ce4ad3e322c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:21:37.259274 20822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ab87b8f82f415d80434ce4ad3e322c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:21:37.259347 21085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ab87b8f82f415d80434ce4ad3e322c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:21:37.271239 21166 tablet_bootstrap.cc:492] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5: Bootstrap starting.
I20260928 02:21:37.274507 20887 heartbeater.cc:503] Master 127.4.140.254:32967 was elected leader, sending a full tablet report...
I20260928 02:21:37.274649 21167 tablet_bootstrap.cc:492] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8: Bootstrap starting.
I20260928 02:21:37.278174 21167 tablet_bootstrap.cc:654] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8: Neither blocks nor log segments found. Creating new log.
I20260928 02:21:37.278983 21166 tablet_bootstrap.cc:654] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5: Neither blocks nor log segments found. Creating new log.
I20260928 02:21:37.279536 21167 log.cc:824] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:21:37.280341 21166 log.cc:824] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:21:37.281685 21168 tablet_bootstrap.cc:492] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031: Bootstrap starting.
I20260928 02:21:37.283390 21167 tablet_bootstrap.cc:492] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8: No bootstrap required, opened a new log
I20260928 02:21:37.283684 21167 ts_tablet_manager.cc:1397] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 02:21:37.285192 21168 tablet_bootstrap.cc:654] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031: Neither blocks nor log segments found. Creating new log.
I20260928 02:21:37.285485 21166 tablet_bootstrap.cc:492] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5: No bootstrap required, opened a new log
I20260928 02:21:37.285759 21166 ts_tablet_manager.cc:1397] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20260928 02:21:37.286387 21168 log.cc:824] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031: Log is configured to *not* fsync() on all Append() calls
I20260928 02:21:37.289399 21168 tablet_bootstrap.cc:492] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031: No bootstrap required, opened a new log
I20260928 02:21:37.289655 21168 ts_tablet_manager.cc:1397] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 02:21:37.293231 21166 raft_consensus.cc:348] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 } } peers { permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 } } peers { permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 } }
I20260928 02:21:37.293691 21166 raft_consensus.cc:374] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:21:37.293797 21166 raft_consensus.cc:729] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e51bb23a4dc4b89ae61691398347cc5, State: Initialized, Role: FOLLOWER
I20260928 02:21:37.294438 21166 consensus_queue.cc:260] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 } } peers { permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 } } peers { permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 } }
I20260928 02:21:37.294364 21167 raft_consensus.cc:348] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 } } peers { permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 } } peers { permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 } }
I20260928 02:21:37.294781 21167 raft_consensus.cc:374] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:21:37.294871 21167 raft_consensus.cc:729] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd09ff4c20d44abab01c6570962c8df8, State: Initialized, Role: FOLLOWER
I20260928 02:21:37.295552 21167 consensus_queue.cc:260] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 } } peers { permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 } } peers { permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 } }
I20260928 02:21:37.295814 21150 heartbeater.cc:503] Master 127.4.140.254:32967 was elected leader, sending a full tablet report...
I20260928 02:21:37.297299 21167 ts_tablet_manager.cc:1428] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260928 02:21:37.297857 21166 ts_tablet_manager.cc:1428] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260928 02:21:37.300482 21168 raft_consensus.cc:348] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 } } peers { permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 } } peers { permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 } }
I20260928 02:21:37.300958 21168 raft_consensus.cc:374] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:21:37.301082 21168 raft_consensus.cc:729] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bb12dd2a4d14e1f9d9470a75e58e031, State: Initialized, Role: FOLLOWER
I20260928 02:21:37.301632 21168 consensus_queue.cc:260] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 } } peers { permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 } } peers { permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 } }
I20260928 02:21:37.302531 21019 heartbeater.cc:503] Master 127.4.140.254:32967 was elected leader, sending a full tablet report...
I20260928 02:21:37.302932 21168 ts_tablet_manager.cc:1428] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260928 02:21:37.335058 21174 raft_consensus.cc:482] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:21:37.335418 21174 raft_consensus.cc:504] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 } } peers { permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 } } peers { permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 } }
I20260928 02:21:37.337257 21174 leader_election.cc:288] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd09ff4c20d44abab01c6570962c8df8 (127.4.140.193:39591), 3e51bb23a4dc4b89ae61691398347cc5 (127.4.140.195:44857)
I20260928 02:21:37.344616 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:37.337829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:49413 (local address 127.4.140.195:44857)
0928 02:21:37.338075 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:37.338084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:37.338743 (+ 659us) server_negotiation.cc:407] Connection header received
0928 02:21:37.338995 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:37.339002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:37.339069 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:37.339170 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:21:37.339691 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.340444 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.342228 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.343397 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.343556 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:37.343629 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:37.344204 (+ 575us) server_negotiation.cc:299] Negotiation successful
0928 02:21:37.344329 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 02:21:37.344719 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:37.337740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:48103 (local address 127.4.140.193:39591)
0928 02:21:37.337974 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:37.337982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:37.339173 (+ 1191us) server_negotiation.cc:407] Connection header received
0928 02:21:37.339369 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:37.339380 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:37.339439 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:37.339532 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:21:37.340116 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.340799 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.342538 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.343722 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.343873 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:37.343967 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:37.344289 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 02:21:37.344422 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:21:37.344769 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:37.337624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:44857 (local address 127.4.140.194:49413)
0928 02:21:37.338616 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:37.338643 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:21:37.338825 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:21:37.339205 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:21:37.339209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:21:37.339237 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:21:37.339551 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:37.339560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.340585 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.340590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:37.342076 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:37.342091 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.343783 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.343792 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:37.344033 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:37.344062 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:21:37.344177 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 02:21:37.344311 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":4,"thread_start_us":565,"threads_started":1}
I20260928 02:21:37.344834 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:37.337899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:39591 (local address 127.4.140.194:48103)
0928 02:21:37.339064 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:37.339080 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:21:37.339233 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:21:37.339550 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:21:37.339555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:21:37.339579 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:21:37.339970 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:37.339980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.340926 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.340934 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:37.342374 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:21:37.342388 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.343977 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.343983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:21:37.344129 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:37.344153 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:21:37.344258 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:21:37.344380 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"mutex_wait_us":5,"thread_start_us":480,"threads_started":1}
I20260928 02:21:37.345780 20842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ab87b8f82f415d80434ce4ad3e322c" candidate_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd09ff4c20d44abab01c6570962c8df8" is_pre_election: true
I20260928 02:21:37.345858 21105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ab87b8f82f415d80434ce4ad3e322c" candidate_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e51bb23a4dc4b89ae61691398347cc5" is_pre_election: true
I20260928 02:21:37.346311 21105 raft_consensus.cc:2454] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bb12dd2a4d14e1f9d9470a75e58e031 in term 0.
I20260928 02:21:37.346303 20842 raft_consensus.cc:2454] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bb12dd2a4d14e1f9d9470a75e58e031 in term 0.
I20260928 02:21:37.346973 20908 leader_election.cc:302] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bb12dd2a4d14e1f9d9470a75e58e031, dd09ff4c20d44abab01c6570962c8df8; no voters:
I20260928 02:21:37.347380 21174 raft_consensus.cc:2781] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:21:37.347532 21174 raft_consensus.cc:482] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:21:37.347651 21174 raft_consensus.cc:3037] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:21:37.349908 21174 raft_consensus.cc:504] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 } } peers { permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 } } peers { permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 } }
I20260928 02:21:37.350555 21174 leader_election.cc:288] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [CANDIDATE]: Term 1 election: Requested vote from peers dd09ff4c20d44abab01c6570962c8df8 (127.4.140.193:39591), 3e51bb23a4dc4b89ae61691398347cc5 (127.4.140.195:44857)
I20260928 02:21:37.350948 20842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ab87b8f82f415d80434ce4ad3e322c" candidate_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd09ff4c20d44abab01c6570962c8df8"
I20260928 02:21:37.351037 21105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ab87b8f82f415d80434ce4ad3e322c" candidate_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e51bb23a4dc4b89ae61691398347cc5"
I20260928 02:21:37.351193 20842 raft_consensus.cc:3037] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:21:37.351221 21105 raft_consensus.cc:3037] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:21:37.353437 20842 raft_consensus.cc:2454] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bb12dd2a4d14e1f9d9470a75e58e031 in term 1.
I20260928 02:21:37.353535 21105 raft_consensus.cc:2454] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bb12dd2a4d14e1f9d9470a75e58e031 in term 1.
I20260928 02:21:37.353932 20908 leader_election.cc:302] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bb12dd2a4d14e1f9d9470a75e58e031, dd09ff4c20d44abab01c6570962c8df8; no voters:
I20260928 02:21:37.354206 21174 raft_consensus.cc:2781] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:21:37.354677 21174 raft_consensus.cc:686] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 1 LEADER]: Becoming Leader. State: Replica: 7bb12dd2a4d14e1f9d9470a75e58e031, State: Running, Role: LEADER
I20260928 02:21:37.355170 21174 consensus_queue.cc:237] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 } } peers { permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 } } peers { permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 } }
I20260928 02:21:37.359938 20702 catalog_manager.cc:5495] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bb12dd2a4d14e1f9d9470a75e58e031 (127.4.140.194). New cstate: current_term: 1 leader_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb12dd2a4d14e1f9d9470a75e58e031" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 } health_report { overall_health: UNKNOWN } } }
I20260928 02:21:37.379650 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:37.373936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34582 (local address 127.4.140.194:37981)
0928 02:21:37.374150 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:37.374156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:37.374284 (+ 128us) server_negotiation.cc:407] Connection header received
0928 02:21:37.374577 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:37.374582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:37.374663 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:37.374830 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:21:37.375207 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.376028 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.377449 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.377873 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.378007 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:37.378529 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:21:37.379035 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:21:37.379195 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:37.379275 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 02:21:37.379355 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:21:37.387876 21105 raft_consensus.cc:1261] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5 [term 1 FOLLOWER]: Refusing update from remote peer 7bb12dd2a4d14e1f9d9470a75e58e031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:21:37.387889 20842 raft_consensus.cc:1261] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8 [term 1 FOLLOWER]: Refusing update from remote peer 7bb12dd2a4d14e1f9d9470a75e58e031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:21:37.388952 21177 consensus_queue.cc:1035] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e51bb23a4dc4b89ae61691398347cc5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 44857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:21:37.389314 21174 consensus_queue.cc:1035] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd09ff4c20d44abab01c6570962c8df8" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:21:37.400202 21180 mvcc.cc:205] Tried to move back new op lower bound from 7334142350889713664 to 7334142350771003392. Current Snapshot: MvccSnapshot[applied={T|T < 7334142350889713664}]
I20260928 02:21:37.416442 20702 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:21:37.416734 20702 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:21:37.420349 20702 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb: Sending DeleteTablet for 3 replicas of tablet 84ab87b8f82f415d80434ce4ad3e322c
I20260928 02:21:37.421514 20954 tablet_service.cc:1514] Processing DeleteTablet for tablet 84ab87b8f82f415d80434ce4ad3e322c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:21:37 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:34570
I20260928 02:21:37.421528 20822 tablet_service.cc:1514] Processing DeleteTablet for tablet 84ab87b8f82f415d80434ce4ad3e322c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:21:37 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:47816
I20260928 02:21:37.422238 21085 tablet_service.cc:1514] Processing DeleteTablet for tablet 84ab87b8f82f415d80434ce4ad3e322c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:21:37 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:32864
I20260928 02:21:37.422427 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 20739
I20260928 02:21:37.422724 21192 tablet_replica.cc:330] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031: stopping tablet replica
I20260928 02:21:37.423277 21192 raft_consensus.cc:2229] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:21:37.424019 21192 raft_consensus.cc:2258] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:21:37.424803 21191 tablet_replica.cc:330] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8: stopping tablet replica
I20260928 02:21:37.425257 21191 raft_consensus.cc:2229] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:21:37.425827 21191 raft_consensus.cc:2258] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:21:37.426244 21192 ts_tablet_manager.cc:1905] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:21:37.427945 21191 ts_tablet_manager.cc:1905] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:21:37.433176 21192 ts_tablet_manager.cc:1918] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:21:37.433347 21192 log.cc:1196] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/84ab87b8f82f415d80434ce4ad3e322c
I20260928 02:21:37.433866 21192 ts_tablet_manager.cc:1939] T 84ab87b8f82f415d80434ce4ad3e322c P 7bb12dd2a4d14e1f9d9470a75e58e031: Deleting consensus metadata
I20260928 02:21:37.434442 21193 tablet_replica.cc:330] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5: stopping tablet replica
I20260928 02:21:37.434881 21193 raft_consensus.cc:2229] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:21:37.435299 21191 ts_tablet_manager.cc:1918] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:21:37.435508 21191 log.cc:1196] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/84ab87b8f82f415d80434ce4ad3e322c
I20260928 02:21:37.435497 21193 raft_consensus.cc:2258] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:21:37.435734 20689 catalog_manager.cc:4841] TS 7bb12dd2a4d14e1f9d9470a75e58e031 (127.4.140.194:37981): tablet 84ab87b8f82f415d80434ce4ad3e322c (table test-table [id=ae79bd05aac24a629500a446cf9beba6]) successfully deleted
I20260928 02:21:37.436275 21191 ts_tablet_manager.cc:1939] T 84ab87b8f82f415d80434ce4ad3e322c P dd09ff4c20d44abab01c6570962c8df8: Deleting consensus metadata
I20260928 02:21:37.437788 20687 catalog_manager.cc:4841] TS dd09ff4c20d44abab01c6570962c8df8 (127.4.140.193:39591): tablet 84ab87b8f82f415d80434ce4ad3e322c (table test-table [id=ae79bd05aac24a629500a446cf9beba6]) successfully deleted
I20260928 02:21:37.437896 21193 ts_tablet_manager.cc:1905] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:21:37.444851 21193 ts_tablet_manager.cc:1918] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:21:37.445039 21193 log.cc:1196] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/84ab87b8f82f415d80434ce4ad3e322c
I20260928 02:21:37.445585 21193 ts_tablet_manager.cc:1939] T 84ab87b8f82f415d80434ce4ad3e322c P 3e51bb23a4dc4b89ae61691398347cc5: Deleting consensus metadata
I20260928 02:21:37.447412 20688 catalog_manager.cc:4841] TS 3e51bb23a4dc4b89ae61691398347cc5 (127.4.140.195:44857): tablet 84ab87b8f82f415d80434ce4ad3e322c (table test-table [id=ae79bd05aac24a629500a446cf9beba6]) successfully deleted
I20260928 02:21:37.448154 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:37.423333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47830 (local address 127.4.140.193:39591)
0928 02:21:37.424022 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:37.424056 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:37.424080 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:21:37.424151 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:37.424159 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:37.424267 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:37.424380 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:21:37.426921 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.428607 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.431237 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.432188 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.432355 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:37.441676 (+ 9321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:21:37.441688 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:21:37.441702 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:21:37.441742 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:21:37.444671 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:37.445512 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:37.445520 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:37.445523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:37.445693 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:37.446093 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:37.446100 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:37.446103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:37.446447 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:21:37.446746 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:37.447084 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 02:21:37.447804 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504}
I20260928 02:21:37.448879 20882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:21:37.543951 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 20739
I20260928 02:21:37.567600 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 20891
I20260928 02:21:37.588665 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:37.568751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.4.140.194:37981)
0928 02:21:37.568916 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:37.568920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:37.568939 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:21:37.568999 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:37.569003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:37.569064 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:37.569156 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:21:37.570869 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.571658 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.573651 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.574087 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.574191 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:37.582485 (+ 8294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:21:37.582499 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:21:37.582516 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:21:37.582558 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:21:37.585596 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:37.586492 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:37.586499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:37.586501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:37.586648 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:37.587032 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:37.587035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:37.587037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:37.587400 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:21:37.587642 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:37.588050 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 02:21:37.588342 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:21:37.589490 21014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:21:37.702421 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 20891
I20260928 02:21:37.721738 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 21022
I20260928 02:21:37.734691 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:37.722458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32872 (local address 127.4.140.195:44857)
0928 02:21:37.722644 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:21:37.722648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:21:37.722666 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:21:37.722755 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:21:37.722759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:21:37.722831 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:21:37.722916 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:21:37.724407 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.725151 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.726374 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:21:37.726756 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:21:37.726846 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:21:37.729512 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:21:37.729527 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:21:37.729542 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:21:37.729578 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:21:37.732390 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:37.733016 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:37.733023 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:37.733025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:37.733159 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:21:37.733480 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:21:37.733483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:21:37.733485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:21:37.733809 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:21:37.734029 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 02:21:37.734287 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 02:21:37.734445 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:21:37.735329 21145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:21:37.824477 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 21022
I20260928 02:21:37.841498 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 20669
I20260928 02:21:37.842691 20732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:21:37.957118 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 20669
I20260928 02:21:57.980438 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:32967
--webserver_interface=127.4.140.254
--webserver_port=40115
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:32967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:21:58.319101 21198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:21:58.319414 21198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:21:58.319607 21198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:21:58.327802 21198 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:21:58.327980 21198 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:21:58.328071 21198 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:21:58.328153 21198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:21:58.328262 21198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:21:58.328394 21198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:21:58.328485 21198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:21:58.328570 21198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:21:58.339238 21198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:32967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:32967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=40115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:21:58.340687 21198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:21:58.342103 21198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:21:58.354436 21203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:58.354594 21204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:58.354620 21206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:21:58.355851 21198 server_base.cc:1034] running on GCE node
I20260928 02:21:58.356775 21198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:21:58.358203 21198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:21:58.359474 21198 hybrid_clock.cc:648] HybridClock initialized: now 1790562118359390 us; error 46 us; skew 500 ppm
I20260928 02:21:58.359875 21198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:21:58 dist-test-slave-h37r krb5kdc[20120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562118, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:21:58.369952 21198 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:21:58.371582 21198 webserver.cc:460] Webserver started at http://127.4.140.254:40115/ using document root <none> and password file <none>
I20260928 02:21:58.372354 21198 fs_manager.cc:359] Metadata directory not provided
I20260928 02:21:58.372449 21198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:21:58.377909 21198 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 02:21:58.380790 21213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:58.382257 21198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:21:58.382395 21198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "6f3cfae4688a4df3bce475f9158ac2bb"
format_stamp: "Formatted at 2026-09-28 02:21:12 on dist-test-slave-h37r"
server_key: "2191e889c53695688ed10ac79da3fc5c"
server_key_iv: "2e0ad528d61e3705115c7c170c3ced9e"
server_key_version: "encryptionkey@0"
I20260928 02:21:58.383333 21198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:21:58.402455 21198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:21:58.406893 21198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:21:58.407150 21198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:21:58.427712 21198 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:32967
I20260928 02:21:58.427748 21264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:32967 every 8 connection(s)
I20260928 02:21:58.429432 21198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:21:58.435019 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 21198
I20260928 02:21:58.435997 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:39591
--local_ip_for_outbound_sockets=127.4.140.193
--tserver_master_addrs=127.4.140.254:32967
--webserver_port=42207
--webserver_interface=127.4.140.193
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:21:58.436903 21265 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:21:58.440348 21265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb: Bootstrap starting.
I20260928 02:21:58.461885 21265 log.cc:824] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:21:58.478192 21265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:21:58.478778 21265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb: Bootstrap complete.
I20260928 02:21:58.488339 21265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32967 } }
I20260928 02:21:58.489315 21265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3cfae4688a4df3bce475f9158ac2bb, State: Initialized, Role: FOLLOWER
I20260928 02:21:58.489912 21265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32967 } }
I20260928 02:21:58.490172 21265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:21:58.490301 21265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:21:58.490478 21265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:21:58.494580 21265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32967 } }
I20260928 02:21:58.495096 21265 leader_election.cc:302] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f3cfae4688a4df3bce475f9158ac2bb; no voters:
I20260928 02:21:58.495815 21265 leader_election.cc:288] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 02:21:58.495968 21269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:21:58.497319 21269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [term 3 LEADER]: Becoming Leader. State: Replica: 6f3cfae4688a4df3bce475f9158ac2bb, State: Running, Role: LEADER
I20260928 02:21:58.497978 21269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32967 } }
I20260928 02:21:58.498227 21265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:21:58.500761 21271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f3cfae4688a4df3bce475f9158ac2bb. Latest consensus state: current_term: 3 leader_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32967 } } }
I20260928 02:21:58.501158 21271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [sys.catalog]: This master's current role is: LEADER
I20260928 02:21:58.501746 21270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3cfae4688a4df3bce475f9158ac2bb" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 32967 } } }
I20260928 02:21:58.502099 21270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb [sys.catalog]: This master's current role is: LEADER
I20260928 02:21:58.513347 21275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:21:58.523090 21275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:21:58.524878 21275 catalog_manager.cc:1259] Loaded cluster ID: 1720ff3f24594a0f95df1c88a5911f1d
I20260928 02:21:58.525000 21275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:21:58.529610 21275 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:21:58.531471 21275 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb: Loaded TSK: 0
I20260928 02:21:58.531661 21275 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6f3cfae4688a4df3bce475f9158ac2bb: Loaded TSK: 1
F20260928 02:21:58.533236 21275 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f48e34839ad google::LogMessage::Fail() at ??:0
@ 0x7f48e348688a google::LogMessage::SendToLog() at ??:0
@ 0x7f48e34834de google::LogMessage::Flush() at ??:0
@ 0x7f48e3484a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f48e71e77d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f48e71d673e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f48f8eb61c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f48f8eb5783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f48f8f412a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f48f8f410c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f48f8f8b5e9 std::function<>::operator()() at ??:0
@ 0x7f48f8eb9cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f48f8eb7c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f48f8f3f610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f48f8f3f439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f48e60cf345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f48e60d291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f48e60d2799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f48e60adeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f48e93876db start_thread at ??:0
@ 0x7f48e231d61f clone at ??:0
W20260928 02:21:58.880643 21267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:21:58.880947 21267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:21:58.881044 21267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:21:58.889024 21267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:21:58.889165 21267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:21:58.889284 21267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:21:58.899818 21267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:39591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=42207
--tserver_master_addrs=127.4.140.254:32967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:21:58.901281 21267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:21:58.902585 21267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:21:58.916445 21312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:58.916541 21313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:58.916499 21315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:21:58.917840 21267 server_base.cc:1034] running on GCE node
I20260928 02:21:58.918578 21267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:21:58.919574 21267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:21:58.920784 21267 hybrid_clock.cc:648] HybridClock initialized: now 1790562118920694 us; error 79 us; skew 500 ppm
I20260928 02:21:58.921140 21267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:21:58 dist-test-slave-h37r krb5kdc[20120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562118, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:21:58.924515 21267 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:21:58.926162 21267 webserver.cc:460] Webserver started at http://127.4.140.193:42207/ using document root <none> and password file <none>
I20260928 02:21:58.926908 21267 fs_manager.cc:359] Metadata directory not provided
I20260928 02:21:58.927042 21267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:21:58.932766 21267 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:21:58.935591 21322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:58.937017 21267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:21:58.937207 21267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "dd09ff4c20d44abab01c6570962c8df8"
format_stamp: "Formatted at 2026-09-28 02:21:13 on dist-test-slave-h37r"
server_key: "1362e6cc326596ae57af4dbad5196012"
server_key_iv: "77206a69e16a1db68e35e9cbafd99098"
server_key_version: "encryptionkey@0"
I20260928 02:21:58.938187 21267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:21:58.965253 21267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:21:58.969945 21267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:21:58.970247 21267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:21:58.971310 21267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:21:58.972796 21267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:21:58.972923 21267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:58.973061 21267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:21:58.973155 21267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:59.008339 21267 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:39591
I20260928 02:21:59.008431 21434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:39591 every 8 connection(s)
I20260928 02:21:59.009836 21267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:21:59.012707 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 21267
I20260928 02:21:59.013381 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:37981
--local_ip_for_outbound_sockets=127.4.140.194
--tserver_master_addrs=127.4.140.254:32967
--webserver_port=37725
--webserver_interface=127.4.140.194
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:21:59.015178 21437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:59.013037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.193:39619)
0928 02:21:59.013915 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:59.014225 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":369,"threads_started":1}
W20260928 02:21:59.015945 21435 heartbeater.cc:644] Failed to heartbeat to 127.4.140.254:32967 (0 consecutive failures): Network error: Failed to ping master at 127.4.140.254:32967: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
I20260928 02:21:59.016871 21437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:59.016532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.193:53997)
0928 02:21:59.016689 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:59.016763 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 02:21:59.017704 21437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:59.017378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.193:32999)
0928 02:21:59.017526 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:59.017605 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 02:21:59.017903 21435 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:21:59.345095 21438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:21:59.345379 21438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:21:59.345472 21438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:21:59.352751 21438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:21:59.352864 21438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:21:59.352977 21438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:21:59.362735 21438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:37981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=37725
--tserver_master_addrs=127.4.140.254:32967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:21:59.364037 21438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:21:59.365288 21438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:21:59.378841 21444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:59.378746 21443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:59.379283 21446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:21:59.379561 21438 server_base.cc:1034] running on GCE node
I20260928 02:21:59.380259 21438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:21:59.381296 21438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:21:59.382544 21438 hybrid_clock.cc:648] HybridClock initialized: now 1790562119382482 us; error 55 us; skew 500 ppm
I20260928 02:21:59.382891 21438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:21:59 dist-test-slave-h37r krb5kdc[20120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562119, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:21:59.386318 21438 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:21:59.388036 21438 webserver.cc:460] Webserver started at http://127.4.140.194:37725/ using document root <none> and password file <none>
I20260928 02:21:59.388736 21438 fs_manager.cc:359] Metadata directory not provided
I20260928 02:21:59.388886 21438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:21:59.394379 21438 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:21:59.397501 21453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:59.399113 21438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:21:59.399293 21438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "7bb12dd2a4d14e1f9d9470a75e58e031"
format_stamp: "Formatted at 2026-09-28 02:21:13 on dist-test-slave-h37r"
server_key: "7b89d87cd2c3076431081e3ffb1ffc8f"
server_key_iv: "91ed23319b8c1a56f68a792ccfd25f19"
server_key_version: "encryptionkey@0"
I20260928 02:21:59.400401 21438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:21:59.422818 21438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:21:59.426961 21438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:21:59.427246 21438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:21:59.428260 21438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:21:59.429697 21438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:21:59.429826 21438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:59.429965 21438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:21:59.430063 21438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:59.464254 21438 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:37981
I20260928 02:21:59.464337 21565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:37981 every 8 connection(s)
I20260928 02:21:59.465756 21438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:21:59.468194 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 21438
I20260928 02:21:59.468954 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:44857
--local_ip_for_outbound_sockets=127.4.140.195
--tserver_master_addrs=127.4.140.254:32967
--webserver_port=43961
--webserver_interface=127.4.140.195
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:21:59.470892 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:59.469056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.194:42411)
0928 02:21:59.469817 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:59.470049 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":290,"threads_started":1}
W20260928 02:21:59.471549 21566 heartbeater.cc:644] Failed to heartbeat to 127.4.140.254:32967 (0 consecutive failures): Network error: Failed to ping master at 127.4.140.254:32967: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
I20260928 02:21:59.472396 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:59.472082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.194:51627)
0928 02:21:59.472226 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:59.472295 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:21:59.473212 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:59.472898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.194:44885)
0928 02:21:59.473036 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:59.473115 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 02:21:59.473429 21566 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:21:59.791376 21569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:21:59.791663 21569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:21:59.791832 21569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:21:59.799391 21569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:21:59.799501 21569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:21:59.799680 21569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:21:59.809785 21569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:44857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=43961
--tserver_master_addrs=127.4.140.254:32967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:21:59.811189 21569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:21:59.812566 21569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:21:59.826287 21575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:59.826356 21577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:21:59.826360 21574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:21:59.827209 21569 server_base.cc:1034] running on GCE node
I20260928 02:21:59.827805 21569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:21:59.828863 21569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:21:59.830106 21569 hybrid_clock.cc:648] HybridClock initialized: now 1790562119830062 us; error 58 us; skew 500 ppm
I20260928 02:21:59.830425 21569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:21:59 dist-test-slave-h37r krb5kdc[20120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562119, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:21:59.833640 21569 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:21:59.835098 21569 webserver.cc:460] Webserver started at http://127.4.140.195:43961/ using document root <none> and password file <none>
I20260928 02:21:59.835803 21569 fs_manager.cc:359] Metadata directory not provided
I20260928 02:21:59.835980 21569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:21:59.841297 21569 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:21:59.844154 21584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:59.845502 21569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:21:59.845680 21569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "3e51bb23a4dc4b89ae61691398347cc5"
format_stamp: "Formatted at 2026-09-28 02:21:14 on dist-test-slave-h37r"
server_key: "2282be1320dfbfa766f8d74f257ef412"
server_key_iv: "cf8591c557f40ffdca81db044c199cc9"
server_key_version: "encryptionkey@0"
I20260928 02:21:59.846652 21569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:21:59.863757 21569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:21:59.868854 21569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:21:59.869170 21569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:21:59.870316 21569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:21:59.871774 21569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:21:59.871944 21569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:59.872097 21569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:21:59.872244 21569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:21:59.910616 21569 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:44857
I20260928 02:21:59.910744 21696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:44857 every 8 connection(s)
I20260928 02:21:59.912313 21569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:21:59.913560 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 21569
I20260928 02:21:59.917668 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:59.915813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.195:35611)
0928 02:21:59.916608 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:59.916817 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":278,"threads_started":1}
W20260928 02:21:59.918349 21697 heartbeater.cc:644] Failed to heartbeat to 127.4.140.254:32967 (0 consecutive failures): Network error: Failed to ping master at 127.4.140.254:32967: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
I20260928 02:21:59.919262 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:59.918904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.195:48139)
0928 02:21:59.919079 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:59.919155 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 02:21:59.920143 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:21:59.919771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.195:58227)
0928 02:21:59.919933 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:21:59.920036 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 02:21:59.920472 21697 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:22:00.019309 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:00.018481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.193:48677)
0928 02:22:00.018961 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:00.019057 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":218,"threads_started":1}
I20260928 02:22:00.474838 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:00.473976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.194:37929)
0928 02:22:00.474492 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:00.474590 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":232,"threads_started":1}
I20260928 02:22:00.921957 21703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:00.921120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.195:33345)
0928 02:22:00.921607 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:00.921717 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":215,"threads_started":1}
I20260928 02:22:01.021126 21704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:01.020374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.193:33387)
0928 02:22:01.020850 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:01.020940 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":238,"threads_started":1}
I20260928 02:22:01.476717 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:01.475868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.194:41441)
0928 02:22:01.476439 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:01.476535 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":250,"threads_started":1}
I20260928 02:22:01.923758 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:01.923028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.195:48981)
0928 02:22:01.923491 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:01.923575 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":227,"threads_started":1}
I20260928 02:22:02.022886 21707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:02.022028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.193:39109)
0928 02:22:02.022561 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:02.022659 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":281,"threads_started":1}
I20260928 02:22:02.478440 21708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:02.477723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.194:54135)
0928 02:22:02.478164 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:02.478253 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":242,"threads_started":1}
I20260928 02:22:02.925348 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:02.924673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.195:42075)
0928 02:22:02.925096 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:02.925180 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":232,"threads_started":1}
I20260928 02:22:03.024595 21710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:03.023849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.193:53647)
0928 02:22:03.024310 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:03.024397 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":249,"threads_started":1}
I20260928 02:22:03.480163 21711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:03.479347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.194:59081)
0928 02:22:03.479805 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:03.479889 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":233,"threads_started":1}
I20260928 02:22:03.926852 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:03.926136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.195:54811)
0928 02:22:03.926582 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:03.926661 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":213,"threads_started":1}
I20260928 02:22:04.026485 21713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:04.025614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.193:42161)
0928 02:22:04.026193 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:04.026292 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":280,"threads_started":1}
I20260928 02:22:04.481848 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:04.481112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.194:57313)
0928 02:22:04.481594 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:04.481677 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":214,"threads_started":1}
I20260928 02:22:04.928893 21715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:04.928021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.195:52281)
0928 02:22:04.928601 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:04.928697 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":301,"threads_started":1}
I20260928 02:22:05.028406 21716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:05.027611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.193:42153)
0928 02:22:05.028125 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:05.028238 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":233,"threads_started":1}
I20260928 02:22:05.483670 21717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:05.482941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.194:56775)
0928 02:22:05.483404 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:05.483497 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":243,"threads_started":1}
I20260928 02:22:05.930749 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:05.929841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.195:46983)
0928 02:22:05.930382 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:05.930507 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":273,"threads_started":1}
I20260928 02:22:06.030349 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:06.029454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.193:51945)
0928 02:22:06.030016 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:06.030114 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":249,"threads_started":1}
I20260928 02:22:06.485512 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:06.484730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.194:39927)
0928 02:22:06.485217 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:06.485299 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":217,"threads_started":1}
I20260928 02:22:06.932567 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:06.931727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.195:52287)
0928 02:22:06.932250 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:06.932336 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":260,"threads_started":1}
I20260928 02:22:07.032123 21722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:07.031321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.193:54319)
0928 02:22:07.031784 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:07.031874 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":251,"threads_started":1}
I20260928 02:22:07.487236 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:07.486513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.194:48343)
0928 02:22:07.486959 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:07.487050 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":232,"threads_started":1}
I20260928 02:22:07.934115 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:07.933420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.195:58103)
0928 02:22:07.933859 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:07.933943 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":234,"threads_started":1}
I20260928 02:22:08.033774 21725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:08.033017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.193:44847)
0928 02:22:08.033496 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:08.033595 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":237,"threads_started":1}
I20260928 02:22:08.489236 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:08.488285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.194:44963)
0928 02:22:08.488915 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:08.489001 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":364,"threads_started":1}
I20260928 02:22:08.936250 21727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:08.935171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.195:41575)
0928 02:22:08.935882 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:08.936006 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":354,"threads_started":1}
I20260928 02:22:09.035580 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:09.034732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.193:47335)
0928 02:22:09.035257 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:09.035352 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":282,"threads_started":1}
I20260928 02:22:09.490953 21729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:09.490203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.194:53359)
0928 02:22:09.490683 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:09.490777 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":263,"threads_started":1}
I20260928 02:22:09.937906 21730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:09.937196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.195:35855)
0928 02:22:09.937656 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:09.937740 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":242,"threads_started":1}
I20260928 02:22:10.037338 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:10.036589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.193:46125)
0928 02:22:10.037058 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:10.037158 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":223,"threads_started":1}
I20260928 02:22:10.492844 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:10.491998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.194:56607)
0928 02:22:10.492568 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:10.492666 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":254,"threads_started":1}
I20260928 02:22:10.939720 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:10.938923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.195:42969)
0928 02:22:10.939440 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:10.939542 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":261,"threads_started":1}
I20260928 02:22:11.039106 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:11.038388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.193:33885)
0928 02:22:11.038855 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:11.038944 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":236,"threads_started":1}
I20260928 02:22:11.494601 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:11.493818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.194:36189)
0928 02:22:11.494292 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:11.494433 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":226,"threads_started":1}
I20260928 02:22:11.941759 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:11.940858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.195:34399)
0928 02:22:11.941425 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:11.941544 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":302,"threads_started":1}
I20260928 02:22:12.040977 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:12.040181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.193:60719)
0928 02:22:12.040673 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:12.040773 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":238,"threads_started":1}
I20260928 02:22:12.496438 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:12.495560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.194:54771)
0928 02:22:12.496139 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:12.496242 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":223,"threads_started":1}
I20260928 02:22:12.943614 21739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:12.942677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.195:52203)
0928 02:22:12.943277 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:12.943376 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":271,"threads_started":1}
I20260928 02:22:13.042640 21740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:13.041885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.193:47927)
0928 02:22:13.042363 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:13.042457 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":215,"threads_started":1}
I20260928 02:22:13.498188 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:13.497359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.194:52291)
0928 02:22:13.497837 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:13.497920 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":268,"threads_started":1}
I20260928 02:22:13.945331 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:13.944570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.195:44093)
0928 02:22:13.945040 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:13.945138 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":229,"threads_started":1}
I20260928 02:22:14.044703 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:14.043858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.193:46537)
0928 02:22:14.044394 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:14.044515 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":236,"threads_started":1}
I20260928 02:22:14.500120 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:14.499319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.194:43615)
0928 02:22:14.499749 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:14.499859 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":213,"threads_started":1}
I20260928 02:22:14.916728 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 21267
I20260928 02:22:14.936197 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:14.917871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56534 (local address 127.4.140.193:39591)
0928 02:22:14.918597 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:14.918618 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:14.918670 (+ 52us) server_negotiation.cc:407] Connection header received
0928 02:22:14.920622 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:14.920634 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:14.921093 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:14.921414 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:14.922771 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:14.924087 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:14.925429 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:14.926112 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:14.926247 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:14.929131 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:14.929139 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:14.929157 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:14.929196 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:14.932269 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:14.933179 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:14.933187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:14.933195 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:14.933381 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:14.933846 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:14.933851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:14.933853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:14.934493 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:14.934752 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:14.935277 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 02:22:14.935733 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":335,"threads_started":1}
I20260928 02:22:14.937670 21430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:14.947467 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:14.946664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:32967 (local address 127.4.140.195:50255)
0928 02:22:14.947177 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:14.947285 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.254:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":282,"threads_started":1}
I20260928 02:22:15.024821 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 21267
I20260928 02:22:15.042053 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 21438
I20260928 02:22:15.061172 21748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:15.043098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34228 (local address 127.4.140.194:37981)
0928 02:22:15.043691 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:15.043706 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:15.043745 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:22:15.045545 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:15.045556 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:15.046017 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:15.046388 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:15.047989 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:15.049271 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:15.050852 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:15.051717 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:15.051983 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:15.054913 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:15.054921 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:15.054937 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:15.054976 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:15.057885 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:15.058583 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:15.058592 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:15.058605 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:15.058771 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:15.059159 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:15.059165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:15.059168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:15.059673 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:15.060009 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:15.060470 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 02:22:15.060887 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":301,"threads_started":1}
I20260928 02:22:15.062577 21561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:15.147264 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 21438
I20260928 02:22:15.165729 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 21569
I20260928 02:22:15.183872 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:15.166772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34078 (local address 127.4.140.195:44857)
0928 02:22:15.167418 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:15.167433 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:15.167474 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:22:15.169171 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:15.169181 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:15.169621 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:15.169905 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:15.171321 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:15.172724 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:15.174037 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:15.174694 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:15.174824 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:15.177508 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:15.177516 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:15.177533 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:15.177578 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:15.180801 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:15.181433 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:15.181439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:15.181446 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:15.181605 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:15.181940 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:15.181943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:15.181945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:15.182492 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:15.182738 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:15.183133 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 02:22:15.183578 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":330,"threads_started":1}
I20260928 02:22:15.185385 21692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:15.269454 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 21569
2026-09-28T02:22:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64399 ms)
[----------] 30 tests from SecurityITest (299876 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:22:15 dist-test-slave-h37r krb5kdc[21754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:22:15 dist-test-slave-h37r krb5kdc[21754](info): set up 2 sockets
Sep 28 02:22:15 dist-test-slave-h37r krb5kdc[21754](info): commencing operation
krb5kdc: starting...
W20260928 02:22:16.404493 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:22:16 dist-test-slave-h37r krb5kdc[21754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:22:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:22:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:22:16.665182 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:35281
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:35281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:17.010972 21767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:17.011278 21767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:17.011493 21767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:17.019001 21767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:22:17.019115 21767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:17.019232 21767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:17.019316 21767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:22:17.019409 21767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:22:17.029629 21767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:35281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:35281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:17.031023 21767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:17.032444 21767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:17.044899 21772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:17.044961 21775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:17.045002 21773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:17.045526 21767 server_base.cc:1034] running on GCE node
I20260928 02:22:17.046450 21767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:17.048022 21767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:17.049289 21767 hybrid_clock.cc:648] HybridClock initialized: now 1790562137049236 us; error 41 us; skew 500 ppm
I20260928 02:22:17.049686 21767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:17 dist-test-slave-h37r krb5kdc[21754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562137, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:17.052927 21767 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:22:17.054593 21767 webserver.cc:460] Webserver started at http://127.4.140.254:37167/ using document root <none> and password file <none>
I20260928 02:22:17.055325 21767 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:17.055475 21767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:17.055804 21767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:17.058902 21767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "f036938664ff4b0ab488c328849d3b04"
format_stamp: "Formatted at 2026-09-28 02:22:17 on dist-test-slave-h37r"
server_key: "580266d42691f998be5df6a4358a0ccb"
server_key_iv: "224cb9b48539fe056f782c7167c9daf6"
server_key_version: "encryptionkey@0"
I20260928 02:22:17.059677 21767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "f036938664ff4b0ab488c328849d3b04"
format_stamp: "Formatted at 2026-09-28 02:22:17 on dist-test-slave-h37r"
server_key: "580266d42691f998be5df6a4358a0ccb"
server_key_iv: "224cb9b48539fe056f782c7167c9daf6"
server_key_version: "encryptionkey@0"
I20260928 02:22:17.064982 21767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 02:22:17.068580 21782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:17.070019 21767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:22:17.070220 21767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "f036938664ff4b0ab488c328849d3b04"
format_stamp: "Formatted at 2026-09-28 02:22:17 on dist-test-slave-h37r"
server_key: "580266d42691f998be5df6a4358a0ccb"
server_key_iv: "224cb9b48539fe056f782c7167c9daf6"
server_key_version: "encryptionkey@0"
I20260928 02:22:17.070488 21767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:17.087499 21767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:17.091874 21767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:17.092209 21767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:17.111183 21767 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:35281
I20260928 02:22:17.111191 21833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:35281 every 8 connection(s)
I20260928 02:22:17.112676 21767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:22:17.116459 21834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:17.121011 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 21767
I20260928 02:22:17.121229 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:22:17.122051 4659 external_mini_cluster.cc:1428] Setting key 72284cfe0cbbd3b29477dc8e1fa026e1
I20260928 02:22:17.128391 21834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04: Bootstrap starting.
I20260928 02:22:17.132026 21834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:17.133235 21834 log.cc:824] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:17.136734 21834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04: No bootstrap required, opened a new log
Sep 28 02:22:17 dist-test-slave-h37r krb5kdc[21754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:22:17.144551 21834 raft_consensus.cc:348] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f036938664ff4b0ab488c328849d3b04" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 35281 } }
I20260928 02:22:17.144942 21834 raft_consensus.cc:374] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:17.145078 21834 raft_consensus.cc:729] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f036938664ff4b0ab488c328849d3b04, State: Initialized, Role: FOLLOWER
I20260928 02:22:17.145757 21834 consensus_queue.cc:260] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f036938664ff4b0ab488c328849d3b04" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 35281 } }
I20260928 02:22:17.146020 21834 raft_consensus.cc:388] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:22:17.146193 21834 raft_consensus.cc:482] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:22:17.146446 21834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:17.146976 21837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:17.124116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36030 (local address 127.4.140.254:35281)
0928 02:22:17.124960 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:17.124971 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:17.125016 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:22:17.126760 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:17.126783 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:17.127389 (+ 606us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:17.127807 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:17.129938 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:17.131292 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:17.133177 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:17.134237 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:17.134443 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:17.139164 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:17.139175 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:17.139189 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:17.139233 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:17.142440 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:17.143148 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:17.143154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:17.143164 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:17.143333 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:17.143768 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:17.143772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:17.143774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:17.144359 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:17.144670 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:17.145145 (+ 475us) server_negotiation.cc:299] Negotiation successful
0928 02:22:17.145645 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":400,"threads_started":1}
I20260928 02:22:17.149163 21834 raft_consensus.cc:504] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f036938664ff4b0ab488c328849d3b04" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 35281 } }
I20260928 02:22:17.149663 21834 leader_election.cc:302] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f036938664ff4b0ab488c328849d3b04; no voters:
I20260928 02:22:17.150277 21834 leader_election.cc:288] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:22:17.150425 21839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:17.151131 21839 raft_consensus.cc:686] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04 [term 1 LEADER]: Becoming Leader. State: Replica: f036938664ff4b0ab488c328849d3b04, State: Running, Role: LEADER
I20260928 02:22:17.151687 21839 consensus_queue.cc:237] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f036938664ff4b0ab488c328849d3b04" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 35281 } }
I20260928 02:22:17.152277 21834 sys_catalog.cc:564] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:22:17.154706 21841 sys_catalog.cc:455] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f036938664ff4b0ab488c328849d3b04. Latest consensus state: current_term: 1 leader_uuid: "f036938664ff4b0ab488c328849d3b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f036938664ff4b0ab488c328849d3b04" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 35281 } } }
I20260928 02:22:17.155597 21841 sys_catalog.cc:458] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:17.155300 21840 sys_catalog.cc:455] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f036938664ff4b0ab488c328849d3b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f036938664ff4b0ab488c328849d3b04" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 35281 } } }
I20260928 02:22:17.156203 21840 sys_catalog.cc:458] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:17.158341 21850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:22:17.164845 21850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:22:17.173146 21850 catalog_manager.cc:1347] Generated new cluster ID: 60e1fd69bbcf42469f51ad096f258336
I20260928 02:22:17.173250 21850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:22:17.202479 21850 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:22:17.204457 21850 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:22:17.222371 21850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04: Generated new TSK 0
I20260928 02:22:17.223272 21850 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
I20260928 02:22:17.295639 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:35281
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:17.638100 21860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:17.638378 21860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:17.638475 21860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:17.646258 21860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:17.646365 21860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:17.646479 21860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:22:17.656670 21860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:17.658074 21860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:17.659327 21860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:17.672390 21865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:17.672441 21866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:17.672477 21868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:17.673311 21860 server_base.cc:1034] running on GCE node
I20260928 02:22:17.673906 21860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:17.674963 21860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:17.676198 21860 hybrid_clock.cc:648] HybridClock initialized: now 1790562137676149 us; error 43 us; skew 500 ppm
I20260928 02:22:17.676581 21860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:17 dist-test-slave-h37r krb5kdc[21754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562137, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:17.684212 21860 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:22:17.685787 21860 webserver.cc:460] Webserver started at http://127.4.140.193:34245/ using document root <none> and password file <none>
I20260928 02:22:17.686480 21860 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:17.686627 21860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:17.686950 21860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:17.689878 21860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "c8ad839a75494b72b3adb9094c43b731"
format_stamp: "Formatted at 2026-09-28 02:22:17 on dist-test-slave-h37r"
server_key: "35b5211abc97a6fb9b58c11d1dd327ac"
server_key_iv: "0a66be600169aa53885180b144f60e1e"
server_key_version: "encryptionkey@0"
I20260928 02:22:17.690665 21860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "c8ad839a75494b72b3adb9094c43b731"
format_stamp: "Formatted at 2026-09-28 02:22:17 on dist-test-slave-h37r"
server_key: "35b5211abc97a6fb9b58c11d1dd327ac"
server_key_iv: "0a66be600169aa53885180b144f60e1e"
server_key_version: "encryptionkey@0"
I20260928 02:22:17.695616 21860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:22:17.699074 21875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:17.700484 21860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:22:17.700656 21860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "c8ad839a75494b72b3adb9094c43b731"
format_stamp: "Formatted at 2026-09-28 02:22:17 on dist-test-slave-h37r"
server_key: "35b5211abc97a6fb9b58c11d1dd327ac"
server_key_iv: "0a66be600169aa53885180b144f60e1e"
server_key_version: "encryptionkey@0"
I20260928 02:22:17.700860 21860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:17.711540 21860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:17.715837 21860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:17.716168 21860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:17.717125 21860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:22:17.718434 21860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:22:17.718536 21860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:17.718688 21860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:22:17.718762 21860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:17.752707 21860 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:45105
I20260928 02:22:17.752789 21987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:45105 every 8 connection(s)
I20260928 02:22:17.754099 21860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:22:17.764562 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 21860
I20260928 02:22:17.764802 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:22:17.765566 4659 external_mini_cluster.cc:1428] Setting key 1f9f0b3096bd8cd1b172eb3737f90d86
Sep 28 02:22:17 dist-test-slave-h37r krb5kdc[21754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562137, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:22:17.774506 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:17.756905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:44155 (local address 127.4.140.254:35281)
0928 02:22:17.757390 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:17.757394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:17.758221 (+ 827us) server_negotiation.cc:407] Connection header received
0928 02:22:17.760071 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:17.760077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:17.760175 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:17.760296 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:17.761763 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:17.762498 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:17.764060 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:17.764460 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:17.764615 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:17.768713 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:17.768719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:17.768722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:17.768752 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:17.771220 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:17.772173 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:17.772178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:17.772180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:17.772293 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:17.772756 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:17.772759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:17.772761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:17.773000 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:17.773105 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:17.774026 (+ 921us) server_negotiation.cc:299] Negotiation successful
0928 02:22:17.774191 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":279,"threads_started":1}
I20260928 02:22:17.775995 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:17.757383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:35281 (local address 127.4.140.193:44155)
0928 02:22:17.758060 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:17.758107 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:17.759738 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:17.760525 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:17.760533 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:17.761049 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:22:17.761589 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:17.761605 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:17.762645 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:17.762650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:17.763865 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:17.763877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:17.764694 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:17.764703 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:17.764992 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:17.765690 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:22:17.765725 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:22:17.768508 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:22:17.771361 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:22:17.771373 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:22:17.771391 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:22:17.772057 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:22:17.772401 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:22:17.772408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:22:17.772414 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:22:17.772648 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:22:17.773127 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:22:17.773138 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:22:17.773621 (+ 483us) client_negotiation.cc:769] Sending connection context
0928 02:22:17.773933 (+ 312us) client_negotiation.cc:240] Negotiation successful
0928 02:22:17.774504 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":252,"threads_started":1}
I20260928 02:22:17.777701 21988 heartbeater.cc:347] Connected to a master server at 127.4.140.254:35281
I20260928 02:22:17.778033 21988 heartbeater.cc:464] Registering TS with master...
I20260928 02:22:17.778702 21988 heartbeater.cc:511] Master 127.4.140.254:35281 requested a full tablet report, sending...
I20260928 02:22:17.781010 21799 ts_manager.cc:194] Registered new tserver with Master: c8ad839a75494b72b3adb9094c43b731 (127.4.140.193:45105)
I20260928 02:22:17.783206 21799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:44155
WARNING: no policy specified for kudu/127.4.140.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
I20260928 02:22:17.824424 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:35281
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:18.153475 21994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:18.153877 21994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:18.154003 21994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:18.161912 21994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:18.162026 21994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:18.162142 21994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:22:18.171833 21994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:18.173139 21994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:18.174496 21994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:18.187690 21999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:18.187609 22000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:18.187654 22002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:18.188580 21994 server_base.cc:1034] running on GCE node
I20260928 02:22:18.189185 21994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:18.190162 21994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:18.191424 21994 hybrid_clock.cc:648] HybridClock initialized: now 1790562138191374 us; error 57 us; skew 500 ppm
I20260928 02:22:18.191769 21994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:18 dist-test-slave-h37r krb5kdc[21754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562138, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:18.194873 21994 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:22:18.196483 21994 webserver.cc:460] Webserver started at http://127.4.140.194:34339/ using document root <none> and password file <none>
I20260928 02:22:18.197183 21994 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:18.197324 21994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:18.197629 21994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:18.200541 21994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "7f820db75dd04d0b9f0c4a10a7179257"
format_stamp: "Formatted at 2026-09-28 02:22:18 on dist-test-slave-h37r"
server_key: "47fb31339b3e434bf233213e5f6c9180"
server_key_iv: "52eb4dfa9c5e158d2a9ce15add3327ba"
server_key_version: "encryptionkey@0"
I20260928 02:22:18.201365 21994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "7f820db75dd04d0b9f0c4a10a7179257"
format_stamp: "Formatted at 2026-09-28 02:22:18 on dist-test-slave-h37r"
server_key: "47fb31339b3e434bf233213e5f6c9180"
server_key_iv: "52eb4dfa9c5e158d2a9ce15add3327ba"
server_key_version: "encryptionkey@0"
I20260928 02:22:18.206689 21994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:22:18.210338 22009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:18.211719 21994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:22:18.211939 21994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "7f820db75dd04d0b9f0c4a10a7179257"
format_stamp: "Formatted at 2026-09-28 02:22:18 on dist-test-slave-h37r"
server_key: "47fb31339b3e434bf233213e5f6c9180"
server_key_iv: "52eb4dfa9c5e158d2a9ce15add3327ba"
server_key_version: "encryptionkey@0"
I20260928 02:22:18.212169 21994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:18.234572 21994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:18.238996 21994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:18.239316 21994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:18.240403 21994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:22:18.241869 21994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:22:18.241979 21994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:18.242131 21994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:22:18.242202 21994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:18.277530 21994 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:42839
I20260928 02:22:18.277592 22121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:42839 every 8 connection(s)
I20260928 02:22:18.278973 21994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:22:18.288646 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 21994
I20260928 02:22:18.288851 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:22:18.289318 4659 external_mini_cluster.cc:1428] Setting key 6dd11b19b1146961d8190b147546bbaa
Sep 28 02:22:18 dist-test-slave-h37r krb5kdc[21754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562138, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:22:18.299399 22124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:18.281583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:37515 (local address 127.4.140.254:35281)
0928 02:22:18.282036 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:18.282041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:18.282908 (+ 867us) server_negotiation.cc:407] Connection header received
0928 02:22:18.284911 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:18.284918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:18.285028 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:18.285152 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:18.286795 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.287513 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.289267 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.289747 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.289883 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:18.293446 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:18.293451 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:18.293454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:18.293486 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:18.296237 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:18.297015 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:18.297019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:18.297022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:18.297155 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:18.297613 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:18.297618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:18.297620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:18.297879 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:18.298050 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:18.298776 (+ 726us) server_negotiation.cc:299] Negotiation successful
0928 02:22:18.299024 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":243,"threads_started":1}
I20260928 02:22:18.300472 22125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:18.282008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:35281 (local address 127.4.140.194:37515)
0928 02:22:18.282740 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:18.282778 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:18.284612 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:18.285401 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:18.285410 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:18.285966 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:22:18.286622 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:18.286642 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.287671 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.287682 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:18.289123 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:18.289136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.289984 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.289991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:18.290220 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:18.290963 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:22:18.290994 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:22:18.293212 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:22:18.296384 (+ 3172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:22:18.296394 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:22:18.296413 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:22:18.296866 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:22:18.297269 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:22:18.297275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:22:18.297278 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:22:18.297475 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:22:18.298010 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:22:18.298018 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:22:18.298396 (+ 378us) client_negotiation.cc:769] Sending connection context
0928 02:22:18.298674 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 02:22:18.299228 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":282,"threads_started":1}
I20260928 02:22:18.302007 22122 heartbeater.cc:347] Connected to a master server at 127.4.140.254:35281
I20260928 02:22:18.302369 22122 heartbeater.cc:464] Registering TS with master...
I20260928 02:22:18.303264 22122 heartbeater.cc:511] Master 127.4.140.254:35281 requested a full tablet report, sending...
I20260928 02:22:18.305060 21799 ts_manager.cc:194] Registered new tserver with Master: 7f820db75dd04d0b9f0c4a10a7179257 (127.4.140.194:42839)
I20260928 02:22:18.306372 21799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:37515
WARNING: no policy specified for kudu/127.4.140.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
I20260928 02:22:18.348610 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:35281
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:18.684445 22128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:18.684727 22128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:18.684823 22128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:18.691980 22128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:18.692093 22128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:18.692310 22128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:22:18.702021 22128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:18.703378 22128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:18.704751 22128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:18.718161 22134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:18.718116 22133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:18.718132 22136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:18.718931 22128 server_base.cc:1034] running on GCE node
I20260928 02:22:18.719568 22128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:18.720629 22128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:18.721858 22128 hybrid_clock.cc:648] HybridClock initialized: now 1790562138721819 us; error 47 us; skew 500 ppm
I20260928 02:22:18.722185 22128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:18 dist-test-slave-h37r krb5kdc[21754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562138, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:18.725343 22128 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:22:18.726903 22128 webserver.cc:460] Webserver started at http://127.4.140.195:40707/ using document root <none> and password file <none>
I20260928 02:22:18.727607 22128 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:18.727757 22128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:18.728164 22128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:18.731125 22128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "c041cbff956145af9f31f6da87dac7fc"
format_stamp: "Formatted at 2026-09-28 02:22:18 on dist-test-slave-h37r"
server_key: "d8ce7575f32752c626b33eb1144d9e1a"
server_key_iv: "fdfa290358f10fdcef08ff6419bef0e9"
server_key_version: "encryptionkey@0"
I20260928 02:22:18.731997 22128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "c041cbff956145af9f31f6da87dac7fc"
format_stamp: "Formatted at 2026-09-28 02:22:18 on dist-test-slave-h37r"
server_key: "d8ce7575f32752c626b33eb1144d9e1a"
server_key_iv: "fdfa290358f10fdcef08ff6419bef0e9"
server_key_version: "encryptionkey@0"
I20260928 02:22:18.737450 22128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 02:22:18.741088 22143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:18.742559 22128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:22:18.742772 22128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "c041cbff956145af9f31f6da87dac7fc"
format_stamp: "Formatted at 2026-09-28 02:22:18 on dist-test-slave-h37r"
server_key: "d8ce7575f32752c626b33eb1144d9e1a"
server_key_iv: "fdfa290358f10fdcef08ff6419bef0e9"
server_key_version: "encryptionkey@0"
I20260928 02:22:18.743005 22128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:18.756582 22128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:18.762974 22128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:18.763275 22128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:18.764351 22128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:22:18.765717 22128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:22:18.765789 22128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:18.765949 22128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:22:18.766021 22128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:18.786326 21988 heartbeater.cc:503] Master 127.4.140.254:35281 was elected leader, sending a full tablet report...
I20260928 02:22:18.798982 22128 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:43935
I20260928 02:22:18.799016 22255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:43935 every 8 connection(s)
I20260928 02:22:18.800350 22128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:22:18.804870 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 22128
I20260928 02:22:18.805068 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:22:18.805645 4659 external_mini_cluster.cc:1428] Setting key f2e45f5fd90d78ec0c99149b3e67b430
Sep 28 02:22:18 dist-test-slave-h37r krb5kdc[21754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562138, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:22:18.821167 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:18.802934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:51507 (local address 127.4.140.254:35281)
0928 02:22:18.803371 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:18.803375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:18.804456 (+ 1081us) server_negotiation.cc:407] Connection header received
0928 02:22:18.807065 (+ 2609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:18.807071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:18.807176 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:18.807271 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:18.809236 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.810075 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.811607 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.812071 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.812166 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:18.815442 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:18.815447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:18.815450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:18.815482 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:18.818078 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:18.818788 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:18.818793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:18.818795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:18.818920 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:18.819321 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:18.819329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:18.819334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:18.819610 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:18.819771 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:18.820621 (+ 850us) server_negotiation.cc:299] Negotiation successful
0928 02:22:18.820830 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":211,"threads_started":1}
I20260928 02:22:18.822121 22259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:18.803570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:35281 (local address 127.4.140.195:51507)
0928 02:22:18.804296 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:18.804337 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:18.806732 (+ 2395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:18.807574 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:18.807586 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:18.808274 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:22:18.809078 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:18.809099 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.810209 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.810212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:18.811476 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:18.811488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.812258 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.812263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:18.812456 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:18.813094 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:22:18.813117 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:22:18.815237 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:22:18.818219 (+ 2982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:22:18.818226 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:22:18.818239 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:22:18.818678 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:22:18.819026 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:22:18.819029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:22:18.819031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:22:18.819203 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:22:18.819738 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:22:18.819746 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:22:18.820191 (+ 445us) client_negotiation.cc:769] Sending connection context
0928 02:22:18.820493 (+ 302us) client_negotiation.cc:240] Negotiation successful
0928 02:22:18.821003 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":302,"threads_started":1}
I20260928 02:22:18.823441 22256 heartbeater.cc:347] Connected to a master server at 127.4.140.254:35281
I20260928 02:22:18.823781 22256 heartbeater.cc:464] Registering TS with master...
I20260928 02:22:18.824568 22256 heartbeater.cc:511] Master 127.4.140.254:35281 requested a full tablet report, sending...
I20260928 02:22:18.826006 21799 ts_manager.cc:194] Registered new tserver with Master: c041cbff956145af9f31f6da87dac7fc (127.4.140.195:43935)
I20260928 02:22:18.826856 21799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:51507
I20260928 02:22:18.833607 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:22:18.851418 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:18.838099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.4.140.254:35281)
0928 02:22:18.838269 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:18.838274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:18.838729 (+ 455us) server_negotiation.cc:407] Connection header received
0928 02:22:18.838939 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:18.838943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:18.839031 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:18.839120 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:18.840651 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.841452 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.842897 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.843304 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.843402 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:18.846056 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:18.846066 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:18.846072 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:18.846115 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:18.848940 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:18.849636 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:18.849640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:18.849643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:18.849774 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:18.850127 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:18.850131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:18.850133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:18.850506 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:18.850693 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:18.851023 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 02:22:18.851193 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:22:18.855672 21799 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:22:18.862679 21799 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:22:18.897909 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:18.875422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:43935 (local address 127.0.0.1:41820)
0928 02:22:18.876350 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:18.876410 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:18.876619 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:18.889557 (+ 12938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:18.889567 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:18.889615 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:18.890158 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:18.890175 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.892241 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.892248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:18.894651 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:18.894667 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.896733 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.896740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:18.896958 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:18.897015 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 02:22:18.897187 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 02:22:18.897423 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":358,"threads_started":1}
I20260928 02:22:18.897994 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:18.875534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.4.140.195:43935)
0928 02:22:18.888893 (+ 13359us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:18.888899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:18.888929 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:22:18.889031 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:18.889038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:18.889262 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:18.889513 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:18.890411 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.892131 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.894859 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.896514 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.896649 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:18.896892 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:18.897235 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 02:22:18.897481 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13188,"thread_start_us":13038,"threads_started":1}
I20260928 02:22:18.900555 22273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:18.877210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42932 (local address 127.4.140.193:45105)
0928 02:22:18.888353 (+ 11143us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:18.888366 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:18.890558 (+ 2192us) server_negotiation.cc:407] Connection header received
0928 02:22:18.890683 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:18.890692 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:18.890959 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:18.891182 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:18.892188 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.893609 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.896146 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.899029 (+ 2883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.899182 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:18.899409 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:18.900045 (+ 636us) server_negotiation.cc:299] Negotiation successful
0928 02:22:18.900218 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10899,"thread_start_us":10687,"threads_started":1}
I20260928 02:22:18.900604 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:18.876686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:45105 (local address 127.0.0.1:42932)
0928 02:22:18.890284 (+ 13598us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:18.890304 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:18.890422 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:18.891420 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:18.891425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:18.891468 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:18.891956 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:18.891973 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.893734 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.893739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:18.895963 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:18.895980 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.899559 (+ 3579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.899565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:18.899808 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:18.899832 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:22:18.899965 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 02:22:18.900122 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13452,"mutex_wait_us":60,"thread_start_us":14959,"threads_started":1}
I20260928 02:22:18.900941 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:18.876731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:42839 (local address 127.0.0.1:58380)
0928 02:22:18.891983 (+ 15252us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:18.892012 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:18.892153 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:18.892821 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:18.892826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:18.892852 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:18.893286 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:18.893297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.895151 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.895156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:18.897295 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:18.897312 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.899434 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.899441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:18.900015 (+ 574us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:18.900045 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 02:22:18.900122 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:22:18.900239 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15072,"mutex_wait_us":367,"thread_start_us":13226,"threads_started":1}
I20260928 02:22:18.901145 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:18.877261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58380 (local address 127.4.140.194:42839)
0928 02:22:18.887415 (+ 10154us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:18.887430 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:18.892126 (+ 4696us) server_negotiation.cc:407] Connection header received
0928 02:22:18.892351 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:18.892360 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:18.892606 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:18.892805 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:18.893467 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.894965 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.897483 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.898965 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.899099 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:18.899279 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:18.900630 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0928 02:22:18.900813 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9887,"thread_start_us":10178,"threads_started":1}
I20260928 02:22:18.902767 22191 tablet_service.cc:1467] Processing CreateTablet for tablet c706c1f122b94dd4a58714d3da0cf1d8 (DEFAULT_TABLE table=test-table [id=957df12951784ba89c26c09a4aec7f62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:22:18.904533 22191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c706c1f122b94dd4a58714d3da0cf1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:18.904273 21923 tablet_service.cc:1467] Processing CreateTablet for tablet c706c1f122b94dd4a58714d3da0cf1d8 (DEFAULT_TABLE table=test-table [id=957df12951784ba89c26c09a4aec7f62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:22:18.904706 22057 tablet_service.cc:1467] Processing CreateTablet for tablet c706c1f122b94dd4a58714d3da0cf1d8 (DEFAULT_TABLE table=test-table [id=957df12951784ba89c26c09a4aec7f62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:22:18.905661 21923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c706c1f122b94dd4a58714d3da0cf1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:18.906203 22057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c706c1f122b94dd4a58714d3da0cf1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:18.919345 22275 tablet_bootstrap.cc:492] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731: Bootstrap starting.
I20260928 02:22:18.922786 22275 tablet_bootstrap.cc:654] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:18.923591 22277 tablet_bootstrap.cc:492] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257: Bootstrap starting.
I20260928 02:22:18.923707 22276 tablet_bootstrap.cc:492] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc: Bootstrap starting.
I20260928 02:22:18.924257 22275 log.cc:824] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:18.927341 22276 tablet_bootstrap.cc:654] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:18.928097 22277 tablet_bootstrap.cc:654] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:18.928442 22275 tablet_bootstrap.cc:492] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731: No bootstrap required, opened a new log
I20260928 02:22:18.928754 22275 ts_tablet_manager.cc:1397] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 02:22:18.928783 22276 log.cc:824] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:18.929162 22277 log.cc:824] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:18.931795 22277 tablet_bootstrap.cc:492] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257: No bootstrap required, opened a new log
I20260928 02:22:18.932125 22277 ts_tablet_manager.cc:1397] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260928 02:22:18.934159 22276 tablet_bootstrap.cc:492] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc: No bootstrap required, opened a new log
I20260928 02:22:18.934439 22276 ts_tablet_manager.cc:1397] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260928 02:22:18.936775 22275 raft_consensus.cc:348] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c041cbff956145af9f31f6da87dac7fc" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43935 } } peers { permanent_uuid: "c8ad839a75494b72b3adb9094c43b731" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45105 } } peers { permanent_uuid: "7f820db75dd04d0b9f0c4a10a7179257" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42839 } }
I20260928 02:22:18.937172 22275 raft_consensus.cc:374] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:18.937335 22275 raft_consensus.cc:729] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8ad839a75494b72b3adb9094c43b731, State: Initialized, Role: FOLLOWER
I20260928 02:22:18.938117 22275 consensus_queue.cc:260] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c041cbff956145af9f31f6da87dac7fc" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43935 } } peers { permanent_uuid: "c8ad839a75494b72b3adb9094c43b731" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45105 } } peers { permanent_uuid: "7f820db75dd04d0b9f0c4a10a7179257" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42839 } }
I20260928 02:22:18.941457 22275 ts_tablet_manager.cc:1428] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260928 02:22:18.943003 22277 raft_consensus.cc:348] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c041cbff956145af9f31f6da87dac7fc" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43935 } } peers { permanent_uuid: "c8ad839a75494b72b3adb9094c43b731" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45105 } } peers { permanent_uuid: "7f820db75dd04d0b9f0c4a10a7179257" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42839 } }
I20260928 02:22:18.943467 22277 raft_consensus.cc:374] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:18.943567 22277 raft_consensus.cc:729] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f820db75dd04d0b9f0c4a10a7179257, State: Initialized, Role: FOLLOWER
I20260928 02:22:18.944224 22277 consensus_queue.cc:260] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c041cbff956145af9f31f6da87dac7fc" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43935 } } peers { permanent_uuid: "c8ad839a75494b72b3adb9094c43b731" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45105 } } peers { permanent_uuid: "7f820db75dd04d0b9f0c4a10a7179257" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42839 } }
I20260928 02:22:18.945477 22122 heartbeater.cc:503] Master 127.4.140.254:35281 was elected leader, sending a full tablet report...
I20260928 02:22:18.945655 22276 raft_consensus.cc:348] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c041cbff956145af9f31f6da87dac7fc" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43935 } } peers { permanent_uuid: "c8ad839a75494b72b3adb9094c43b731" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45105 } } peers { permanent_uuid: "7f820db75dd04d0b9f0c4a10a7179257" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42839 } }
I20260928 02:22:18.946089 22276 raft_consensus.cc:374] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:18.946187 22276 raft_consensus.cc:729] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c041cbff956145af9f31f6da87dac7fc, State: Initialized, Role: FOLLOWER
I20260928 02:22:18.946215 22277 ts_tablet_manager.cc:1428] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260928 02:22:18.946846 22276 consensus_queue.cc:260] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c041cbff956145af9f31f6da87dac7fc" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43935 } } peers { permanent_uuid: "c8ad839a75494b72b3adb9094c43b731" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45105 } } peers { permanent_uuid: "7f820db75dd04d0b9f0c4a10a7179257" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42839 } }
I20260928 02:22:18.948014 22282 raft_consensus.cc:482] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:22:18.948410 22282 raft_consensus.cc:504] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c041cbff956145af9f31f6da87dac7fc" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43935 } } peers { permanent_uuid: "c8ad839a75494b72b3adb9094c43b731" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45105 } } peers { permanent_uuid: "7f820db75dd04d0b9f0c4a10a7179257" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42839 } }
I20260928 02:22:18.951820 22256 heartbeater.cc:503] Master 127.4.140.254:35281 was elected leader, sending a full tablet report...
I20260928 02:22:18.952931 22276 ts_tablet_manager.cc:1428] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc: Time spent starting tablet: real 0.018s user 0.014s sys 0.005s
I20260928 02:22:18.957093 22282 leader_election.cc:288] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c041cbff956145af9f31f6da87dac7fc (127.4.140.195:43935), c8ad839a75494b72b3adb9094c43b731 (127.4.140.193:45105)
I20260928 02:22:18.967300 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:18.951458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:54643 (local address 127.4.140.195:43935)
0928 02:22:18.951667 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:18.951677 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:18.960678 (+ 9001us) server_negotiation.cc:407] Connection header received
0928 02:22:18.961409 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:18.961415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:18.961496 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:18.961610 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:18.962248 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.963212 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.965370 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.966212 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.966339 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:18.966394 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:18.966872 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 02:22:18.967017 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:22:18.967595 22285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:18.951114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:43935 (local address 127.4.140.194:54643)
0928 02:22:18.960568 (+ 9454us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:18.960592 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:18.961203 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:18.961639 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:18.961643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:18.961672 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:18.962098 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:18.962113 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.963360 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.963365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:18.965202 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:18.965215 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.966444 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.966454 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:18.966652 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:18.966731 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 02:22:18.966844 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 02:22:18.967191 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9271,"mutex_wait_us":278,"spinlock_wait_cycles":10112,"thread_start_us":9055,"threads_started":1}
I20260928 02:22:18.967767 22273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:18.960910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:34607 (local address 127.4.140.193:45105)
0928 02:22:18.961099 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:18.961103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:18.961656 (+ 553us) server_negotiation.cc:407] Connection header received
0928 02:22:18.961937 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:18.961944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:18.962031 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:18.962135 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:18.962805 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.963721 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.965544 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.966382 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.966550 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:18.966623 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:18.967384 (+ 761us) server_negotiation.cc:299] Negotiation successful
0928 02:22:18.967521 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:22:18.968608 22286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:18.960711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:45105 (local address 127.4.140.194:34607)
0928 02:22:18.961541 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:18.961576 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:18.961765 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:18.962151 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:18.962154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:18.962198 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:18.962609 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:18.962622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.963869 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.963873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:18.965381 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:18.965395 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:18.966648 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:18.966653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:18.967176 (+ 523us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:18.967222 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 02:22:18.968046 (+ 824us) client_negotiation.cc:240] Negotiation successful
0928 02:22:18.968217 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":40,"thread_start_us":504,"threads_started":1}
I20260928 02:22:18.968745 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c706c1f122b94dd4a58714d3da0cf1d8" candidate_uuid: "7f820db75dd04d0b9f0c4a10a7179257" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c041cbff956145af9f31f6da87dac7fc" is_pre_election: true
I20260928 02:22:18.969372 22211 raft_consensus.cc:2454] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f820db75dd04d0b9f0c4a10a7179257 in term 0.
I20260928 02:22:18.969594 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c706c1f122b94dd4a58714d3da0cf1d8" candidate_uuid: "7f820db75dd04d0b9f0c4a10a7179257" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8ad839a75494b72b3adb9094c43b731" is_pre_election: true
I20260928 02:22:18.970148 22012 leader_election.cc:302] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f820db75dd04d0b9f0c4a10a7179257, c041cbff956145af9f31f6da87dac7fc; no voters:
I20260928 02:22:18.970158 21943 raft_consensus.cc:2454] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f820db75dd04d0b9f0c4a10a7179257 in term 0.
I20260928 02:22:18.970548 22284 raft_consensus.cc:2781] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:22:18.970733 22284 raft_consensus.cc:482] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:22:18.970880 22284 raft_consensus.cc:3037] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:18.973272 22284 raft_consensus.cc:504] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c041cbff956145af9f31f6da87dac7fc" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43935 } } peers { permanent_uuid: "c8ad839a75494b72b3adb9094c43b731" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45105 } } peers { permanent_uuid: "7f820db75dd04d0b9f0c4a10a7179257" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42839 } }
I20260928 02:22:18.974000 22284 leader_election.cc:288] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [CANDIDATE]: Term 1 election: Requested vote from peers c041cbff956145af9f31f6da87dac7fc (127.4.140.195:43935), c8ad839a75494b72b3adb9094c43b731 (127.4.140.193:45105)
I20260928 02:22:18.974406 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c706c1f122b94dd4a58714d3da0cf1d8" candidate_uuid: "7f820db75dd04d0b9f0c4a10a7179257" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c041cbff956145af9f31f6da87dac7fc"
I20260928 02:22:18.974457 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c706c1f122b94dd4a58714d3da0cf1d8" candidate_uuid: "7f820db75dd04d0b9f0c4a10a7179257" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8ad839a75494b72b3adb9094c43b731"
I20260928 02:22:18.974628 22211 raft_consensus.cc:3037] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:18.974691 21943 raft_consensus.cc:3037] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:18.977988 21943 raft_consensus.cc:2454] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f820db75dd04d0b9f0c4a10a7179257 in term 1.
I20260928 02:22:18.977988 22211 raft_consensus.cc:2454] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f820db75dd04d0b9f0c4a10a7179257 in term 1.
I20260928 02:22:18.978401 22012 leader_election.cc:302] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f820db75dd04d0b9f0c4a10a7179257, c041cbff956145af9f31f6da87dac7fc; no voters:
I20260928 02:22:18.978677 22284 raft_consensus.cc:2781] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:18.978910 22284 raft_consensus.cc:686] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [term 1 LEADER]: Becoming Leader. State: Replica: 7f820db75dd04d0b9f0c4a10a7179257, State: Running, Role: LEADER
I20260928 02:22:18.979382 22284 consensus_queue.cc:237] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c041cbff956145af9f31f6da87dac7fc" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43935 } } peers { permanent_uuid: "c8ad839a75494b72b3adb9094c43b731" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45105 } } peers { permanent_uuid: "7f820db75dd04d0b9f0c4a10a7179257" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42839 } }
I20260928 02:22:18.984402 21798 catalog_manager.cc:5495] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f820db75dd04d0b9f0c4a10a7179257 (127.4.140.194). New cstate: current_term: 1 leader_uuid: "7f820db75dd04d0b9f0c4a10a7179257" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c041cbff956145af9f31f6da87dac7fc" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8ad839a75494b72b3adb9094c43b731" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f820db75dd04d0b9f0c4a10a7179257" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42839 } health_report { overall_health: HEALTHY } } }
W20260928 02:22:19.006608 21989 tablet.cc:2369] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:22:19.024426 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:19.019039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58394 (local address 127.4.140.194:42839)
0928 02:22:19.019328 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:19.019340 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:19.019357 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:22:19.019458 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:19.019462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:19.019580 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:19.019715 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:22:19.020158 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:19.020878 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:19.022302 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:19.022715 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:19.022807 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:19.023251 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:22:19.023790 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:22:19.023957 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:19.024040 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 02:22:19.024109 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
W20260928 02:22:19.031068 22123 tablet.cc:2369] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:22:19.032805 22211 raft_consensus.cc:1261] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc [term 1 FOLLOWER]: Refusing update from remote peer 7f820db75dd04d0b9f0c4a10a7179257: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:22:19.032821 21943 raft_consensus.cc:1261] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731 [term 1 FOLLOWER]: Refusing update from remote peer 7f820db75dd04d0b9f0c4a10a7179257: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:22:19.033847 22284 consensus_queue.cc:1035] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c041cbff956145af9f31f6da87dac7fc" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:22:19.034197 22282 consensus_queue.cc:1035] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ad839a75494b72b3adb9094c43b731" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:22:19.049270 22290 mvcc.cc:205] Tried to move back new op lower bound from 7334142521467654144 to 7334142521263755264. Current Snapshot: MvccSnapshot[applied={T|T < 7334142521467654144}]
I20260928 02:22:19.060308 21798 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:22:19.060595 21798 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:22:19.064126 21798 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f036938664ff4b0ab488c328849d3b04: Sending DeleteTablet for 3 replicas of tablet c706c1f122b94dd4a58714d3da0cf1d8
I20260928 02:22:19.065348 22191 tablet_service.cc:1514] Processing DeleteTablet for tablet c706c1f122b94dd4a58714d3da0cf1d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:22:19 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:41820
I20260928 02:22:19.065387 21923 tablet_service.cc:1514] Processing DeleteTablet for tablet c706c1f122b94dd4a58714d3da0cf1d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:22:19 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:42932
I20260928 02:22:19.065802 22057 tablet_service.cc:1514] Processing DeleteTablet for tablet c706c1f122b94dd4a58714d3da0cf1d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:22:19 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:58380
I20260928 02:22:19.067574 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 21860
I20260928 02:22:19.077927 22300 tablet_replica.cc:330] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257: stopping tablet replica
I20260928 02:22:19.078562 22300 raft_consensus.cc:2229] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:22:19.079633 22300 raft_consensus.cc:2258] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:22:19.081734 22298 tablet_replica.cc:330] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc: stopping tablet replica
I20260928 02:22:19.082330 22299 tablet_replica.cc:330] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731: stopping tablet replica
I20260928 02:22:19.082867 22299 raft_consensus.cc:2229] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:22:19.082979 22300 ts_tablet_manager.cc:1905] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:22:19.083446 22298 raft_consensus.cc:2229] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:22:19.084312 22298 raft_consensus.cc:2258] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:22:19.084640 22299 raft_consensus.cc:2258] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:22:19.086938 22298 ts_tablet_manager.cc:1905] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:22:19.089010 22299 ts_tablet_manager.cc:1905] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:22:19.090660 22300 ts_tablet_manager.cc:1918] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:22:19.090878 22300 log.cc:1196] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/c706c1f122b94dd4a58714d3da0cf1d8
I20260928 02:22:19.091472 22300 ts_tablet_manager.cc:1939] T c706c1f122b94dd4a58714d3da0cf1d8 P 7f820db75dd04d0b9f0c4a10a7179257: Deleting consensus metadata
Sep 28 02:22:19 dist-test-slave-h37r krb5kdc[21754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:22:19.093554 21786 catalog_manager.cc:4841] TS 7f820db75dd04d0b9f0c4a10a7179257 (127.4.140.194:42839): tablet c706c1f122b94dd4a58714d3da0cf1d8 (table test-table [id=957df12951784ba89c26c09a4aec7f62]) successfully deleted
I20260928 02:22:19.098829 22299 ts_tablet_manager.cc:1918] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:22:19.099141 22299 log.cc:1196] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/c706c1f122b94dd4a58714d3da0cf1d8
I20260928 02:22:19.099529 22298 ts_tablet_manager.cc:1918] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:22:19.099727 22298 log.cc:1196] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/c706c1f122b94dd4a58714d3da0cf1d8
I20260928 02:22:19.099879 22299 ts_tablet_manager.cc:1939] T c706c1f122b94dd4a58714d3da0cf1d8 P c8ad839a75494b72b3adb9094c43b731: Deleting consensus metadata
I20260928 02:22:19.100327 22298 ts_tablet_manager.cc:1939] T c706c1f122b94dd4a58714d3da0cf1d8 P c041cbff956145af9f31f6da87dac7fc: Deleting consensus metadata
I20260928 02:22:19.101281 22273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:19.071743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42944 (local address 127.4.140.193:45105)
0928 02:22:19.071997 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:19.072006 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:19.080888 (+ 8882us) server_negotiation.cc:407] Connection header received
0928 02:22:19.081135 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:19.081141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:19.081236 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:19.081350 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:19.083408 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:19.084566 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:19.086650 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:19.087271 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:19.087370 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:19.093125 (+ 5755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:19.093139 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:19.093157 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:19.093202 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:19.097430 (+ 4228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:19.098409 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:19.098418 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:19.098422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:19.098621 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:19.099151 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:19.099159 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:19.099165 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:19.099700 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:19.100023 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:19.100651 (+ 628us) server_negotiation.cc:299] Negotiation successful
0928 02:22:19.100877 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:22:19.102110 21785 catalog_manager.cc:4841] TS c8ad839a75494b72b3adb9094c43b731 (127.4.140.193:45105): tablet c706c1f122b94dd4a58714d3da0cf1d8 (table test-table [id=957df12951784ba89c26c09a4aec7f62]) successfully deleted
I20260928 02:22:19.102398 21785 catalog_manager.cc:4841] TS c041cbff956145af9f31f6da87dac7fc (127.4.140.195:43935): tablet c706c1f122b94dd4a58714d3da0cf1d8 (table test-table [id=957df12951784ba89c26c09a4aec7f62]) successfully deleted
I20260928 02:22:19.102882 21983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:19.218173 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 21860
I20260928 02:22:19.236449 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 21994
Sep 28 02:22:19 dist-test-slave-h37r krb5kdc[21754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.194@KRBTEST.COM
I20260928 02:22:19.254784 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:19.237279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58406 (local address 127.4.140.194:42839)
0928 02:22:19.237486 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:19.237492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:19.237514 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:22:19.237691 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:19.237697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:19.237780 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:19.237898 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:19.239284 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:19.240580 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:19.241817 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:19.242478 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:19.242592 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:19.247304 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:19.247325 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:19.247348 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:19.247429 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:19.251445 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:19.252465 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:19.252475 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:19.252479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:19.252706 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:19.253116 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:19.253122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:19.253126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:19.253621 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:19.253967 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:19.254111 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 02:22:19.254353 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:22:19.255584 22117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:19.370564 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 21994
I20260928 02:22:19.390856 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 22128
Sep 28 02:22:19 dist-test-slave-h37r krb5kdc[21754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.195@KRBTEST.COM
I20260928 02:22:19.406454 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:19.391588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41830 (local address 127.4.140.195:43935)
0928 02:22:19.391750 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:19.391754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:19.391772 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:22:19.392060 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:19.392064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:19.392130 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:19.392225 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:19.393619 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:19.394364 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:19.395614 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:19.396086 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:19.396182 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:19.400615 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:19.400630 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:19.400645 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:19.400680 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:19.403845 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:19.404597 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:19.404604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:19.404606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:19.404755 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:19.405090 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:19.405093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:19.405095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:19.405449 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:19.405685 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:19.406008 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 02:22:19.406201 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:22:19.407140 22251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:19.495779 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 22128
I20260928 02:22:19.513013 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 21767
I20260928 02:22:19.514111 21829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:19.623521 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 21767
2026-09-28T02:22:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4357 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T02:22:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:22:19Z Disabled control of system clock
I20260928 02:22:19.700815 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:43061
--webserver_interface=127.4.140.254
--webserver_port=0
--builtin_ntp_servers=127.4.140.212:36971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:43061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:22:20.035656 22309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:20.036067 22309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:20.036159 22309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:20.044114 22309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:22:20.044245 22309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:20.044284 22309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:20.044343 22309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:22:20.044387 22309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:22:20.054121 22309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:43061
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:43061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:20.055420 22309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:20.057258 22309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:20.069576 22315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:20.069576 22314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:20.070176 22317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:20.070699 22309 server_base.cc:1034] running on GCE node
I20260928 02:22:20.071551 22309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:20.073215 22309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:20.074455 22309 hybrid_clock.cc:648] HybridClock initialized: now 1790562140074406 us; error 64 us; skew 500 ppm
I20260928 02:22:20.074815 22309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:22:20.077148 22309 webserver.cc:460] Webserver started at http://127.4.140.254:44827/ using document root <none> and password file <none>
I20260928 02:22:20.077857 22309 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:20.077991 22309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:20.078310 22309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:20.081275 22309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "c60367c419e543269d3a198953f6fd7b"
format_stamp: "Formatted at 2026-09-28 02:22:20 on dist-test-slave-h37r"
server_key: "ca7c6b55611f358c18da5e06ad7d8f6b"
server_key_iv: "3f6b23d9f5b2d0a36fae186898afffb2"
server_key_version: "encryptionkey@0"
I20260928 02:22:20.082147 22309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "c60367c419e543269d3a198953f6fd7b"
format_stamp: "Formatted at 2026-09-28 02:22:20 on dist-test-slave-h37r"
server_key: "ca7c6b55611f358c18da5e06ad7d8f6b"
server_key_iv: "3f6b23d9f5b2d0a36fae186898afffb2"
server_key_version: "encryptionkey@0"
I20260928 02:22:20.087697 22309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:22:20.091217 22323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:20.092723 22309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:22:20.092926 22309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "c60367c419e543269d3a198953f6fd7b"
format_stamp: "Formatted at 2026-09-28 02:22:20 on dist-test-slave-h37r"
server_key: "ca7c6b55611f358c18da5e06ad7d8f6b"
server_key_iv: "3f6b23d9f5b2d0a36fae186898afffb2"
server_key_version: "encryptionkey@0"
I20260928 02:22:20.093169 22309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:20.125557 22309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:20.126482 22309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:20.126789 22309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:20.145728 22309 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:43061
I20260928 02:22:20.145750 22374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:43061 every 8 connection(s)
I20260928 02:22:20.147140 22309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:22:20.150923 22375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:20.155030 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 22309
I20260928 02:22:20.155244 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:22:20.155735 4659 external_mini_cluster.cc:1428] Setting key e056417f4b351fa632f0742c8757a541
I20260928 02:22:20.161880 22375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b: Bootstrap starting.
I20260928 02:22:20.164984 22375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:20.166026 22375 log.cc:824] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:20.169199 22375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b: No bootstrap required, opened a new log
I20260928 02:22:20.169653 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:20.157613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51392 (local address 127.4.140.254:43061)
0928 02:22:20.158609 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:20.158621 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:20.158661 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:22:20.160660 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:20.160684 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:20.161157 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:20.161561 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:20.162130 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:20.163453 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:20.164962 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:20.165751 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:20.165964 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:20.167174 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:20.167187 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:20.167212 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:20.167819 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:20.167951 (+ 132us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:20.167983 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:20.168078 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:20.168316 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 02:22:20.168665 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":344,"threads_started":1}
I20260928 02:22:20.176751 22375 raft_consensus.cc:348] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60367c419e543269d3a198953f6fd7b" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43061 } }
I20260928 02:22:20.177127 22375 raft_consensus.cc:374] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:20.177238 22375 raft_consensus.cc:729] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60367c419e543269d3a198953f6fd7b, State: Initialized, Role: FOLLOWER
I20260928 02:22:20.177799 22375 consensus_queue.cc:260] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60367c419e543269d3a198953f6fd7b" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43061 } }
I20260928 02:22:20.178052 22375 raft_consensus.cc:388] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:22:20.178197 22375 raft_consensus.cc:482] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:22:20.178371 22375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:20.180680 22375 raft_consensus.cc:504] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60367c419e543269d3a198953f6fd7b" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43061 } }
I20260928 02:22:20.181173 22375 leader_election.cc:302] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c60367c419e543269d3a198953f6fd7b; no voters:
I20260928 02:22:20.181743 22375 leader_election.cc:288] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:22:20.181874 22380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:20.182648 22380 raft_consensus.cc:686] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b [term 1 LEADER]: Becoming Leader. State: Replica: c60367c419e543269d3a198953f6fd7b, State: Running, Role: LEADER
I20260928 02:22:20.183246 22380 consensus_queue.cc:237] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60367c419e543269d3a198953f6fd7b" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43061 } }
I20260928 02:22:20.183849 22375 sys_catalog.cc:564] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:22:20.186079 22381 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c60367c419e543269d3a198953f6fd7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60367c419e543269d3a198953f6fd7b" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43061 } } }
I20260928 02:22:20.186631 22381 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:20.186105 22382 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c60367c419e543269d3a198953f6fd7b. Latest consensus state: current_term: 1 leader_uuid: "c60367c419e543269d3a198953f6fd7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60367c419e543269d3a198953f6fd7b" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43061 } } }
I20260928 02:22:20.186947 22382 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:20.188817 22385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:22:20.194008 22385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:22:20.202878 22385 catalog_manager.cc:1347] Generated new cluster ID: 254e414de4a045f0849ef352e96f6811
I20260928 02:22:20.203011 22385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:22:20.220463 22385 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:22:20.222345 22385 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:22:20.233069 22385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b: Generated new TSK 0
I20260928 02:22:20.234165 22385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:22:20.244902 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43061
--builtin_ntp_servers=127.4.140.212:36971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:22:20.580150 22399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:20.580466 22399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:20.580627 22399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:20.588220 22399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:20.588341 22399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:20.588510 22399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:22:20.598126 22399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:20.599409 22399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:20.601231 22399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:20.614094 22404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:20.614054 22407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:20.614014 22405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:20.614975 22399 server_base.cc:1034] running on GCE node
I20260928 02:22:20.615670 22399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:20.616698 22399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:20.617929 22399 hybrid_clock.cc:648] HybridClock initialized: now 1790562140617884 us; error 59 us; skew 500 ppm
I20260928 02:22:20.618247 22399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:22:20.620441 22399 webserver.cc:460] Webserver started at http://127.4.140.193:37133/ using document root <none> and password file <none>
I20260928 02:22:20.621125 22399 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:20.621258 22399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:20.621582 22399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:20.624429 22399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "4b24a31d748c41628bbb4aca5acd7408"
format_stamp: "Formatted at 2026-09-28 02:22:20 on dist-test-slave-h37r"
server_key: "2ceac6aa7a91f8acde14ae83ee18e04c"
server_key_iv: "913a89bd404be9fe3de0962be9b1fcaf"
server_key_version: "encryptionkey@0"
I20260928 02:22:20.625213 22399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "4b24a31d748c41628bbb4aca5acd7408"
format_stamp: "Formatted at 2026-09-28 02:22:20 on dist-test-slave-h37r"
server_key: "2ceac6aa7a91f8acde14ae83ee18e04c"
server_key_iv: "913a89bd404be9fe3de0962be9b1fcaf"
server_key_version: "encryptionkey@0"
I20260928 02:22:20.630074 22399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:22:20.633414 22413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:20.634693 22399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:22:20.634864 22399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "4b24a31d748c41628bbb4aca5acd7408"
format_stamp: "Formatted at 2026-09-28 02:22:20 on dist-test-slave-h37r"
server_key: "2ceac6aa7a91f8acde14ae83ee18e04c"
server_key_iv: "913a89bd404be9fe3de0962be9b1fcaf"
server_key_version: "encryptionkey@0"
I20260928 02:22:20.635107 22399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:20.656677 22399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:20.657562 22399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:20.657879 22399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:20.658864 22399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:22:20.660431 22399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:22:20.660578 22399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:20.660718 22399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:22:20.660809 22399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:20.694705 22399 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:43991
I20260928 02:22:20.694736 22525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:43991 every 8 connection(s)
I20260928 02:22:20.696171 22399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:22:20.702880 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 22399
I20260928 02:22:20.703084 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:22:20.707242 4659 external_mini_cluster.cc:1428] Setting key 06c0ec8050bbd286f43e84a9c432ca66
I20260928 02:22:20.709416 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:20.698913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:57489 (local address 127.4.140.254:43061)
0928 02:22:20.699320 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:20.699325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:20.700654 (+ 1329us) server_negotiation.cc:407] Connection header received
0928 02:22:20.702497 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:20.702502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:20.702618 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:20.702735 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:20.703937 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:20.704684 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:20.706265 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:20.706671 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:20.706817 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:20.708563 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:20.708568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:20.708571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:20.708589 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:20.708631 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:20.708644 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:20.708742 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:20.708968 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 02:22:20.709069 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":194,"threads_started":1}
I20260928 02:22:20.710179 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43061
--builtin_ntp_servers=127.4.140.212:36971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:22:20.710494 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:20.699563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43061 (local address 127.4.140.193:57489)
0928 02:22:20.700481 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:20.700522 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:20.702208 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:20.702975 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:20.702983 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:20.703119 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:22:20.703745 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:20.703762 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:20.704856 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:20.704865 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:20.706110 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:20.706122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:20.706915 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:20.706921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:20.707157 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:20.708181 (+ 1024us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:22:20.708191 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:22:20.708283 (+ 92us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:22:20.708369 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:22:20.708740 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:22:20.708753 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:22:20.708763 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:22:20.708959 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 02:22:20.709369 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":305,"threads_started":1}
I20260928 02:22:20.712031 22526 heartbeater.cc:347] Connected to a master server at 127.4.140.254:43061
I20260928 02:22:20.712337 22526 heartbeater.cc:464] Registering TS with master...
I20260928 02:22:20.713069 22526 heartbeater.cc:511] Master 127.4.140.254:43061 requested a full tablet report, sending...
I20260928 02:22:20.715349 22340 ts_manager.cc:194] Registered new tserver with Master: 4b24a31d748c41628bbb4aca5acd7408 (127.4.140.193:43991)
I20260928 02:22:20.717514 22340 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.140.193:57489
W20260928 02:22:21.051373 22530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:21.051716 22530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:21.051878 22530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:21.059309 22530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:21.059417 22530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:21.059541 22530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:22:21.068663 22530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:21.069903 22530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:21.071578 22530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:21.084410 22535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:21.084373 22536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:21.084502 22538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:21.085125 22530 server_base.cc:1034] running on GCE node
I20260928 02:22:21.085755 22530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:21.086701 22530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:21.087998 22530 hybrid_clock.cc:648] HybridClock initialized: now 1790562141087930 us; error 59 us; skew 500 ppm
I20260928 02:22:21.088338 22530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:22:21.090724 22530 webserver.cc:460] Webserver started at http://127.4.140.194:37285/ using document root <none> and password file <none>
I20260928 02:22:21.091367 22530 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:21.091451 22530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:21.091724 22530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:21.094549 22530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "3ea846cb83504c0d9fff39506273b465"
format_stamp: "Formatted at 2026-09-28 02:22:21 on dist-test-slave-h37r"
server_key: "61e6fde8a0d3a901de36df912c7bac96"
server_key_iv: "6b3a469ba3d21a1e6d97d6a7bc7a931c"
server_key_version: "encryptionkey@0"
I20260928 02:22:21.095285 22530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "3ea846cb83504c0d9fff39506273b465"
format_stamp: "Formatted at 2026-09-28 02:22:21 on dist-test-slave-h37r"
server_key: "61e6fde8a0d3a901de36df912c7bac96"
server_key_iv: "6b3a469ba3d21a1e6d97d6a7bc7a931c"
server_key_version: "encryptionkey@0"
I20260928 02:22:21.100356 22530 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 02:22:21.103758 22544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:21.105134 22530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:22:21.105295 22530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "3ea846cb83504c0d9fff39506273b465"
format_stamp: "Formatted at 2026-09-28 02:22:21 on dist-test-slave-h37r"
server_key: "61e6fde8a0d3a901de36df912c7bac96"
server_key_iv: "6b3a469ba3d21a1e6d97d6a7bc7a931c"
server_key_version: "encryptionkey@0"
I20260928 02:22:21.105520 22530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:21.137220 22530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:21.138136 22530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:21.138475 22530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:21.139573 22530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:22:21.141263 22530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:22:21.141350 22530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:21.141510 22530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:22:21.141609 22530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:21.175839 22530 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:39843
I20260928 02:22:21.175956 22656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:39843 every 8 connection(s)
I20260928 02:22:21.177188 22530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:22:21.187454 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 22530
I20260928 02:22:21.187649 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:22:21.188221 4659 external_mini_cluster.cc:1428] Setting key 4bccd7c28af9832bf41cf5bb065186bc
I20260928 02:22:21.189713 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:21.179906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:43721 (local address 127.4.140.254:43061)
0928 02:22:21.180102 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:21.180107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:21.181432 (+ 1325us) server_negotiation.cc:407] Connection header received
0928 02:22:21.183195 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:21.183199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:21.183266 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:21.183356 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:21.184645 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.185342 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.186796 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.187190 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.187336 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:21.188862 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:21.188866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:21.188869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:21.188887 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:21.188916 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:21.188927 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:21.189065 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:21.189348 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 02:22:21.189425 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:22:21.191028 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:21.180440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43061 (local address 127.4.140.194:43721)
0928 02:22:21.181306 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:21.181345 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:21.182919 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:21.183558 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:21.183565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:21.183651 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:22:21.184461 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:21.184480 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.185460 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.185463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:21.186667 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:21.186679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.187435 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.187443 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:21.187733 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:21.188540 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:22:21.188543 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:22:21.188581 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:22:21.188656 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:22:21.189065 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:22:21.189080 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:22:21.189090 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:22:21.189304 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 02:22:21.189681 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":254,"threads_started":1}
I20260928 02:22:21.191314 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43061
--builtin_ntp_servers=127.4.140.212:36971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:22:21.192519 22657 heartbeater.cc:347] Connected to a master server at 127.4.140.254:43061
I20260928 02:22:21.192823 22657 heartbeater.cc:464] Registering TS with master...
I20260928 02:22:21.193557 22657 heartbeater.cc:511] Master 127.4.140.254:43061 requested a full tablet report, sending...
I20260928 02:22:21.194963 22340 ts_manager.cc:194] Registered new tserver with Master: 3ea846cb83504c0d9fff39506273b465 (127.4.140.194:39843)
I20260928 02:22:21.196255 22340 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.140.194:43721
W20260928 02:22:21.517725 22660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:21.517999 22660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:21.518092 22660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:21.525260 22660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:21.525357 22660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:21.525462 22660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:22:21.534351 22660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:21.535629 22660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:21.537361 22660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:21.550643 22665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:21.550719 22666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:21.550683 22668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:21.551591 22660 server_base.cc:1034] running on GCE node
I20260928 02:22:21.552278 22660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:21.553274 22660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:21.554510 22660 hybrid_clock.cc:648] HybridClock initialized: now 1790562141554457 us; error 47 us; skew 500 ppm
I20260928 02:22:21.554873 22660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:22:21.557155 22660 webserver.cc:460] Webserver started at http://127.4.140.195:43963/ using document root <none> and password file <none>
I20260928 02:22:21.557860 22660 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:21.557992 22660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:21.558322 22660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:21.561259 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "fb4a771128a0464184b83809013b9930"
format_stamp: "Formatted at 2026-09-28 02:22:21 on dist-test-slave-h37r"
server_key: "44e729d7bd23c4a7cf6766a2787cc4cf"
server_key_iv: "de49e33e1a50caebb05649e955d41d5e"
server_key_version: "encryptionkey@0"
I20260928 02:22:21.562053 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "fb4a771128a0464184b83809013b9930"
format_stamp: "Formatted at 2026-09-28 02:22:21 on dist-test-slave-h37r"
server_key: "44e729d7bd23c4a7cf6766a2787cc4cf"
server_key_iv: "de49e33e1a50caebb05649e955d41d5e"
server_key_version: "encryptionkey@0"
I20260928 02:22:21.567117 22660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:22:21.570546 22674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:21.571774 22660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:22:21.571962 22660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "fb4a771128a0464184b83809013b9930"
format_stamp: "Formatted at 2026-09-28 02:22:21 on dist-test-slave-h37r"
server_key: "44e729d7bd23c4a7cf6766a2787cc4cf"
server_key_iv: "de49e33e1a50caebb05649e955d41d5e"
server_key_version: "encryptionkey@0"
I20260928 02:22:21.572237 22660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:21.595724 22660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:21.596717 22660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:21.597040 22660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:21.598129 22660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:22:21.599627 22660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:22:21.599756 22660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:21.599890 22660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:22:21.600003 22660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 02:22:21.632658 22660 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:33695
I20260928 02:22:21.632746 22786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:33695 every 8 connection(s)
I20260928 02:22:21.633967 22660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:22:21.635565 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 22660
I20260928 02:22:21.635746 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:22:21.636319 4659 external_mini_cluster.cc:1428] Setting key 6ecd03fd9709ee8de54d4c885256eee5
I20260928 02:22:21.647213 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:21.636721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:53501 (local address 127.4.140.254:43061)
0928 02:22:21.636947 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:21.636954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:21.638661 (+ 1707us) server_negotiation.cc:407] Connection header received
0928 02:22:21.640754 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:21.640760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:21.640845 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:21.640987 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:21.642081 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.642826 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.644494 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.644906 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.645055 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:21.646357 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:21.646366 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:21.646375 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:21.646395 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:21.646426 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:21.646438 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:21.646534 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:21.646771 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 02:22:21.646869 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:22:21.648375 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:21.637374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43061 (local address 127.4.140.195:53501)
0928 02:22:21.638470 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:21.638558 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:21.640478 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:21.641223 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:21.641232 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:21.641333 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:22:21.641904 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:21.641920 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.642983 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.642988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:21.644350 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:21.644362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.645152 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.645159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:21.645390 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:21.646041 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:22:21.646045 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:22:21.646084 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:22:21.646165 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:22:21.646532 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:22:21.646565 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:22:21.646578 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:22:21.646758 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 02:22:21.647161 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":294,"threads_started":1}
I20260928 02:22:21.649761 22787 heartbeater.cc:347] Connected to a master server at 127.4.140.254:43061
I20260928 02:22:21.650081 22787 heartbeater.cc:464] Registering TS with master...
I20260928 02:22:21.650934 22787 heartbeater.cc:511] Master 127.4.140.254:43061 requested a full tablet report, sending...
I20260928 02:22:21.652484 22340 ts_manager.cc:194] Registered new tserver with Master: fb4a771128a0464184b83809013b9930 (127.4.140.195:33695)
I20260928 02:22:21.653226 22340 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.140.195:53501
I20260928 02:22:21.663781 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:22:21.674098 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:21.667844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.4.140.254:43061)
0928 02:22:21.668169 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:21.668177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:21.668486 (+ 309us) server_negotiation.cc:407] Connection header received
0928 02:22:21.668743 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:21.668747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:21.668822 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:21.668928 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:21.669425 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.670150 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.671573 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.672116 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.672264 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:21.673318 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:21.673326 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:21.673329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:21.673348 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:21.673381 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:21.673393 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:21.673489 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:21.673655 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 02:22:21.673753 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 02:22:21.678297 22340 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:22:21.685457 22340 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:22:21.712337 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:21.698561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:39843 (local address 127.0.0.1:42248)
0928 02:22:21.701031 (+ 2470us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:21.701094 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:21.701294 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:21.702441 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:21.702451 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:21.702491 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:21.702910 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:21.702926 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.705242 (+ 2316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.705251 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:21.708193 (+ 2942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:21.708215 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.711125 (+ 2910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.711134 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:21.711414 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:21.711485 (+ 71us) client_negotiation.cc:769] Sending connection context
0928 02:22:21.711638 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 02:22:21.711883 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2153,"threads_started":1}
I20260928 02:22:21.713445 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:21.699105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42248 (local address 127.4.140.194:39843)
0928 02:22:21.699618 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:21.699630 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:21.701503 (+ 1873us) server_negotiation.cc:407] Connection header received
0928 02:22:21.701648 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:21.701659 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:21.701934 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:21.702136 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:21.703128 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.705038 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.708553 (+ 3515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.710494 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.710662 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:21.710975 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:21.712792 (+ 1817us) server_negotiation.cc:299] Negotiation successful
0928 02:22:21.713024 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":266,"threads_started":1}
I20260928 02:22:21.721190 22526 heartbeater.cc:503] Master 127.4.140.254:43061 was elected leader, sending a full tablet report...
I20260928 02:22:21.722137 22592 tablet_service.cc:1467] Processing CreateTablet for tablet 1a2dbb27fba34d8c98f9bf3a465f4515 (DEFAULT_TABLE table=test-table [id=40fbe746b8b14a0f8da146ea0689d5ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:22:21.723269 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:21.698697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:43991 (local address 127.0.0.1:34478)
0928 02:22:21.713928 (+ 15231us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:21.713954 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:21.714089 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:21.715271 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:21.715279 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:21.715317 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:21.715594 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:21.715607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.717991 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.717995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:21.719485 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:21.719499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.722165 (+ 2666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.722176 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:21.722436 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:21.722471 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:22:21.722573 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:22:21.722911 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15064,"mutex_wait_us":1,"thread_start_us":18553,"threads_started":1}
I20260928 02:22:21.723630 22592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a2dbb27fba34d8c98f9bf3a465f4515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:21.724187 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:21.699253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34478 (local address 127.4.140.193:43991)
0928 02:22:21.713706 (+ 14453us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:21.713718 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:21.714474 (+ 756us) server_negotiation.cc:407] Connection header received
0928 02:22:21.714622 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:21.714635 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:21.714982 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:21.715811 (+ 829us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:21.716153 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.717836 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.719653 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.721612 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.721749 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:21.722065 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:21.723618 (+ 1553us) server_negotiation.cc:299] Negotiation successful
0928 02:22:21.723779 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14218,"thread_start_us":14081,"threads_started":1}
I20260928 02:22:21.726569 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:21.699772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:33695 (local address 127.0.0.1:55134)
0928 02:22:21.717049 (+ 17277us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:21.717073 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:21.717176 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:21.719439 (+ 2263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:21.719443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:21.719468 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:21.719883 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:21.719924 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.721783 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.721788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:21.723949 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:21.723965 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.725770 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.725777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:21.725983 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:21.726016 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:22:21.726114 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 02:22:21.726240 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17142,"mutex_wait_us":29,"thread_start_us":799,"threads_started":1}
I20260928 02:22:21.726706 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:21.700052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55134 (local address 127.4.140.195:33695)
0928 02:22:21.718612 (+ 18560us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:21.718618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:21.718646 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:22:21.718721 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:21.718726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:21.718956 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:21.719157 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:21.720153 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.721629 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.724179 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.725552 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.725706 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:21.725950 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:21.726202 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 02:22:21.726376 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18406,"thread_start_us":18262,"threads_started":1}
I20260928 02:22:21.727742 22461 tablet_service.cc:1467] Processing CreateTablet for tablet 1a2dbb27fba34d8c98f9bf3a465f4515 (DEFAULT_TABLE table=test-table [id=40fbe746b8b14a0f8da146ea0689d5ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:22:21.729107 22461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a2dbb27fba34d8c98f9bf3a465f4515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:21.731285 22722 tablet_service.cc:1467] Processing CreateTablet for tablet 1a2dbb27fba34d8c98f9bf3a465f4515 (DEFAULT_TABLE table=test-table [id=40fbe746b8b14a0f8da146ea0689d5ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:22:21.732681 22722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a2dbb27fba34d8c98f9bf3a465f4515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:21.739660 22805 tablet_bootstrap.cc:492] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465: Bootstrap starting.
I20260928 02:22:21.744681 22806 tablet_bootstrap.cc:492] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 4b24a31d748c41628bbb4aca5acd7408: Bootstrap starting.
I20260928 02:22:21.745648 22805 tablet_bootstrap.cc:654] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:21.746102 22807 tablet_bootstrap.cc:492] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930: Bootstrap starting.
I20260928 02:22:21.747105 22805 log.cc:824] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:21.748413 22806 tablet_bootstrap.cc:654] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 4b24a31d748c41628bbb4aca5acd7408: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:21.749504 22806 log.cc:824] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 4b24a31d748c41628bbb4aca5acd7408: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:21.750470 22805 tablet_bootstrap.cc:492] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465: No bootstrap required, opened a new log
I20260928 02:22:21.750689 22807 tablet_bootstrap.cc:654] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:21.750782 22805 ts_tablet_manager.cc:1397] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260928 02:22:21.751992 22807 log.cc:824] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:21.752206 22806 tablet_bootstrap.cc:492] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 4b24a31d748c41628bbb4aca5acd7408: No bootstrap required, opened a new log
I20260928 02:22:21.752496 22806 ts_tablet_manager.cc:1397] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 4b24a31d748c41628bbb4aca5acd7408: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 02:22:21.755029 22807 tablet_bootstrap.cc:492] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930: No bootstrap required, opened a new log
I20260928 02:22:21.755308 22807 ts_tablet_manager.cc:1397] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260928 02:22:21.759644 22806 raft_consensus.cc:348] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 4b24a31d748c41628bbb4aca5acd7408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea846cb83504c0d9fff39506273b465" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39843 } } peers { permanent_uuid: "4b24a31d748c41628bbb4aca5acd7408" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 43991 } } peers { permanent_uuid: "fb4a771128a0464184b83809013b9930" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33695 } }
I20260928 02:22:21.760077 22806 raft_consensus.cc:374] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 4b24a31d748c41628bbb4aca5acd7408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:21.760201 22806 raft_consensus.cc:729] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 4b24a31d748c41628bbb4aca5acd7408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b24a31d748c41628bbb4aca5acd7408, State: Initialized, Role: FOLLOWER
I20260928 02:22:21.760954 22806 consensus_queue.cc:260] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 4b24a31d748c41628bbb4aca5acd7408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea846cb83504c0d9fff39506273b465" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39843 } } peers { permanent_uuid: "4b24a31d748c41628bbb4aca5acd7408" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 43991 } } peers { permanent_uuid: "fb4a771128a0464184b83809013b9930" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33695 } }
I20260928 02:22:21.761309 22805 raft_consensus.cc:348] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea846cb83504c0d9fff39506273b465" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39843 } } peers { permanent_uuid: "4b24a31d748c41628bbb4aca5acd7408" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 43991 } } peers { permanent_uuid: "fb4a771128a0464184b83809013b9930" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33695 } }
I20260928 02:22:21.761791 22805 raft_consensus.cc:374] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:21.761922 22805 raft_consensus.cc:729] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ea846cb83504c0d9fff39506273b465, State: Initialized, Role: FOLLOWER
I20260928 02:22:21.762564 22805 consensus_queue.cc:260] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea846cb83504c0d9fff39506273b465" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39843 } } peers { permanent_uuid: "4b24a31d748c41628bbb4aca5acd7408" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 43991 } } peers { permanent_uuid: "fb4a771128a0464184b83809013b9930" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33695 } }
I20260928 02:22:21.763054 22806 ts_tablet_manager.cc:1428] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 4b24a31d748c41628bbb4aca5acd7408: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260928 02:22:21.764045 22657 heartbeater.cc:503] Master 127.4.140.254:43061 was elected leader, sending a full tablet report...
I20260928 02:22:21.765601 22805 ts_tablet_manager.cc:1428] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260928 02:22:21.766271 22807 raft_consensus.cc:348] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea846cb83504c0d9fff39506273b465" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39843 } } peers { permanent_uuid: "4b24a31d748c41628bbb4aca5acd7408" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 43991 } } peers { permanent_uuid: "fb4a771128a0464184b83809013b9930" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33695 } }
I20260928 02:22:21.766779 22807 raft_consensus.cc:374] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:21.766889 22807 raft_consensus.cc:729] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4a771128a0464184b83809013b9930, State: Initialized, Role: FOLLOWER
I20260928 02:22:21.767639 22807 consensus_queue.cc:260] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea846cb83504c0d9fff39506273b465" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39843 } } peers { permanent_uuid: "4b24a31d748c41628bbb4aca5acd7408" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 43991 } } peers { permanent_uuid: "fb4a771128a0464184b83809013b9930" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33695 } }
I20260928 02:22:21.768751 22787 heartbeater.cc:503] Master 127.4.140.254:43061 was elected leader, sending a full tablet report...
I20260928 02:22:21.769359 22807 ts_tablet_manager.cc:1428] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260928 02:22:21.835106 22813 raft_consensus.cc:482] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:22:21.835460 22813 raft_consensus.cc:504] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea846cb83504c0d9fff39506273b465" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39843 } } peers { permanent_uuid: "4b24a31d748c41628bbb4aca5acd7408" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 43991 } } peers { permanent_uuid: "fb4a771128a0464184b83809013b9930" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33695 } }
I20260928 02:22:21.837217 22813 leader_election.cc:288] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ea846cb83504c0d9fff39506273b465 (127.4.140.194:39843), 4b24a31d748c41628bbb4aca5acd7408 (127.4.140.193:43991)
I20260928 02:22:21.843865 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:21.837558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:55037 (local address 127.4.140.194:39843)
0928 02:22:21.838032 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:21.838055 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:21.838077 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:22:21.838159 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:21.838166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:21.838278 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:21.838382 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:21.838887 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.839970 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.841892 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.843050 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.843166 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:21.843215 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:21.843576 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 02:22:21.843666 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296}
I20260928 02:22:21.844013 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:21.837448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:39843 (local address 127.4.140.195:55037)
0928 02:22:21.837754 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:21.837774 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:21.837907 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:21.838430 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:21.838442 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:21.838469 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:21.838674 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:21.838705 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.840107 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.840113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:21.841715 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:21.841726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.843198 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.843204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:21.843401 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:21.843436 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:22:21.843541 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:22:21.843685 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":1}
I20260928 02:22:21.844846 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:21.837656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:43991 (local address 127.4.140.195:59557)
0928 02:22:21.838555 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:21.838568 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:21.838671 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:21.839132 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:21.839138 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:21.839183 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:21.839528 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:21.839555 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.840910 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.840914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:21.842657 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:21.842673 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.844243 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.844251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:21.844417 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:21.844446 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:22:21.844556 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:22:21.844655 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":539,"threads_started":1}
I20260928 02:22:21.844998 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:21.837771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:59557 (local address 127.4.140.193:43991)
0928 02:22:21.838020 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:21.838030 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:21.838665 (+ 635us) server_negotiation.cc:407] Connection header received
0928 02:22:21.838851 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:21.838860 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:21.838986 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:21.839080 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:21.839690 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.840758 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.842848 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.843968 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.844134 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:21.844211 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:21.844607 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 02:22:21.844753 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:22:21.845170 22612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a2dbb27fba34d8c98f9bf3a465f4515" candidate_uuid: "fb4a771128a0464184b83809013b9930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ea846cb83504c0d9fff39506273b465" is_pre_election: true
I20260928 02:22:21.845717 22612 raft_consensus.cc:2454] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb4a771128a0464184b83809013b9930 in term 0.
I20260928 02:22:21.845844 22481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a2dbb27fba34d8c98f9bf3a465f4515" candidate_uuid: "fb4a771128a0464184b83809013b9930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b24a31d748c41628bbb4aca5acd7408" is_pre_election: true
I20260928 02:22:21.846391 22481 raft_consensus.cc:2454] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 4b24a31d748c41628bbb4aca5acd7408 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb4a771128a0464184b83809013b9930 in term 0.
I20260928 02:22:21.846408 22675 leader_election.cc:302] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ea846cb83504c0d9fff39506273b465, fb4a771128a0464184b83809013b9930; no voters:
I20260928 02:22:21.846791 22813 raft_consensus.cc:2781] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:22:21.846971 22813 raft_consensus.cc:482] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:22:21.847079 22813 raft_consensus.cc:3037] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:21.849280 22813 raft_consensus.cc:504] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea846cb83504c0d9fff39506273b465" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39843 } } peers { permanent_uuid: "4b24a31d748c41628bbb4aca5acd7408" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 43991 } } peers { permanent_uuid: "fb4a771128a0464184b83809013b9930" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33695 } }
I20260928 02:22:21.849831 22813 leader_election.cc:288] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [CANDIDATE]: Term 1 election: Requested vote from peers 3ea846cb83504c0d9fff39506273b465 (127.4.140.194:39843), 4b24a31d748c41628bbb4aca5acd7408 (127.4.140.193:43991)
I20260928 02:22:21.850250 22612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a2dbb27fba34d8c98f9bf3a465f4515" candidate_uuid: "fb4a771128a0464184b83809013b9930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ea846cb83504c0d9fff39506273b465"
I20260928 02:22:21.850286 22481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a2dbb27fba34d8c98f9bf3a465f4515" candidate_uuid: "fb4a771128a0464184b83809013b9930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b24a31d748c41628bbb4aca5acd7408"
I20260928 02:22:21.850466 22481 raft_consensus.cc:3037] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 4b24a31d748c41628bbb4aca5acd7408 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:21.850476 22612 raft_consensus.cc:3037] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:21.853112 22481 raft_consensus.cc:2454] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 4b24a31d748c41628bbb4aca5acd7408 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb4a771128a0464184b83809013b9930 in term 1.
I20260928 02:22:21.853247 22612 raft_consensus.cc:2454] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb4a771128a0464184b83809013b9930 in term 1.
I20260928 02:22:21.853562 22676 leader_election.cc:302] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b24a31d748c41628bbb4aca5acd7408, fb4a771128a0464184b83809013b9930; no voters:
I20260928 02:22:21.853945 22813 raft_consensus.cc:2781] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:21.854410 22813 raft_consensus.cc:686] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [term 1 LEADER]: Becoming Leader. State: Replica: fb4a771128a0464184b83809013b9930, State: Running, Role: LEADER
I20260928 02:22:21.854978 22813 consensus_queue.cc:237] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea846cb83504c0d9fff39506273b465" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39843 } } peers { permanent_uuid: "4b24a31d748c41628bbb4aca5acd7408" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 43991 } } peers { permanent_uuid: "fb4a771128a0464184b83809013b9930" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33695 } }
I20260928 02:22:21.860198 22340 catalog_manager.cc:5495] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb4a771128a0464184b83809013b9930 (127.4.140.195). New cstate: current_term: 1 leader_uuid: "fb4a771128a0464184b83809013b9930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea846cb83504c0d9fff39506273b465" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b24a31d748c41628bbb4aca5acd7408" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 43991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb4a771128a0464184b83809013b9930" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33695 } health_report { overall_health: HEALTHY } } }
W20260928 02:22:21.885619 22788 tablet.cc:2369] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:22:21.901718 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:21.895448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.4.140.195:33695)
0928 02:22:21.895664 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:21.895693 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:21.895713 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:22:21.895966 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:21.895972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:21.896076 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:21.896270 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:22:21.896731 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.897617 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.899583 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.900054 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.900177 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:21.900676 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:22:21.901168 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:22:21.901331 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:21.901381 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 02:22:21.901457 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:22:21.910175 22481 raft_consensus.cc:1261] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 4b24a31d748c41628bbb4aca5acd7408 [term 1 FOLLOWER]: Refusing update from remote peer fb4a771128a0464184b83809013b9930: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:22:21.910176 22612 raft_consensus.cc:1261] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465 [term 1 FOLLOWER]: Refusing update from remote peer fb4a771128a0464184b83809013b9930: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:22:21.911316 22813 consensus_queue.cc:1035] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ea846cb83504c0d9fff39506273b465" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:22:21.911643 22815 consensus_queue.cc:1035] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b24a31d748c41628bbb4aca5acd7408" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 43991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:22:21.937566 22339 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:22:21.937821 22339 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:22:21.941630 22339 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c60367c419e543269d3a198953f6fd7b: Sending DeleteTablet for 3 replicas of tablet 1a2dbb27fba34d8c98f9bf3a465f4515
I20260928 02:22:21.942905 22592 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a2dbb27fba34d8c98f9bf3a465f4515 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:22:21 UTC) from {username='slave'} at 127.0.0.1:42248
I20260928 02:22:21.942910 22461 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a2dbb27fba34d8c98f9bf3a465f4515 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:22:21 UTC) from {username='slave'} at 127.0.0.1:34478
I20260928 02:22:21.943293 22722 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a2dbb27fba34d8c98f9bf3a465f4515 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:22:21 UTC) from {username='slave'} at 127.0.0.1:55134
I20260928 02:22:21.945402 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 22399
I20260928 02:22:21.956975 22829 tablet_replica.cc:330] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930: stopping tablet replica
I20260928 02:22:21.958645 22829 raft_consensus.cc:2229] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:22:21.959697 22829 raft_consensus.cc:2258] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:22:21.968533 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:21.946693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.4.140.193:43991)
0928 02:22:21.946868 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:21.946876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:21.958164 (+ 11288us) server_negotiation.cc:407] Connection header received
0928 02:22:21.958397 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:21.958403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:21.958531 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:21.958666 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:21.959342 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.960641 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.963654 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:21.964370 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:21.964476 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:21.966417 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:21.966429 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:21.966449 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:21.967332 (+ 883us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:21.967420 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:21.967448 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:21.967534 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:21.968019 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 02:22:21.968147 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:22:21.970034 22521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:21.972321 22827 tablet_replica.cc:330] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465: stopping tablet replica
I20260928 02:22:21.975728 22827 raft_consensus.cc:2229] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:22:21.976686 22827 raft_consensus.cc:2258] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:22:21.977295 22829 ts_tablet_manager.cc:1905] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:22:21.979801 22827 ts_tablet_manager.cc:1905] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:22:21.986857 22827 ts_tablet_manager.cc:1918] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:22:21.987052 22827 log.cc:1196] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/1a2dbb27fba34d8c98f9bf3a465f4515
I20260928 02:22:21.987370 22829 ts_tablet_manager.cc:1918] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:22:21.987619 22827 ts_tablet_manager.cc:1939] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 3ea846cb83504c0d9fff39506273b465: Deleting consensus metadata
I20260928 02:22:21.987643 22829 log.cc:1196] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/1a2dbb27fba34d8c98f9bf3a465f4515
I20260928 02:22:21.988433 22829 ts_tablet_manager.cc:1939] T 1a2dbb27fba34d8c98f9bf3a465f4515 P fb4a771128a0464184b83809013b9930: Deleting consensus metadata
I20260928 02:22:21.991173 22326 catalog_manager.cc:4841] TS fb4a771128a0464184b83809013b9930 (127.4.140.195:33695): tablet 1a2dbb27fba34d8c98f9bf3a465f4515 (table test-table [id=40fbe746b8b14a0f8da146ea0689d5ba]) successfully deleted
I20260928 02:22:21.993999 22324 catalog_manager.cc:4841] TS 3ea846cb83504c0d9fff39506273b465 (127.4.140.194:39843): tablet 1a2dbb27fba34d8c98f9bf3a465f4515 (table test-table [id=40fbe746b8b14a0f8da146ea0689d5ba]) successfully deleted
I20260928 02:22:22.166744 22828 tablet_replica.cc:330] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 4b24a31d748c41628bbb4aca5acd7408: stopping tablet replica
I20260928 02:22:22.167279 22828 raft_consensus.cc:2229] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 4b24a31d748c41628bbb4aca5acd7408 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:22:22.167840 22828 raft_consensus.cc:2258] T 1a2dbb27fba34d8c98f9bf3a465f4515 P 4b24a31d748c41628bbb4aca5acd7408 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:22:22.168838 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 22399
W20260928 02:22:22.194905 22325 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.140.193:43991 (error 108)
I20260928 02:22:22.195639 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 22530
I20260928 02:22:22.206233 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:22.196728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42250 (local address 127.4.140.194:39843)
0928 02:22:22.196936 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:22.196943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:22.197685 (+ 742us) server_negotiation.cc:407] Connection header received
0928 02:22:22.197758 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:22.197764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:22.197865 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:22.197966 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:22.198596 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:22.199727 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:22.201623 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:22.202261 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:22.202398 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:22.203982 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:22.203995 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:22.204016 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:22.204827 (+ 811us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:22.204893 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:22.204921 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:22.205044 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:22.205706 (+ 662us) server_negotiation.cc:299] Negotiation successful
0928 02:22:22.205825 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:22:22.207252 22652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:22.217969 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:22.217363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:43991 (local address 127.0.0.1:34492)
0928 02:22:22.217581 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:22.217728 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:43991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 02:22:22.218384 22325 catalog_manager.cc:4584] TS 4b24a31d748c41628bbb4aca5acd7408 (127.4.140.193:43991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a2dbb27fba34d8c98f9bf3a465f4515: Network error: Client connection negotiation failed: client connection to 127.4.140.193:43991: connect: Connection refused (error 111)
I20260928 02:22:22.272483 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:22.271764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:43991 (local address 127.0.0.1:34496)
0928 02:22:22.272130 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:22.272248 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:43991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
I20260928 02:22:22.273895 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:22.273333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:43991 (local address 127.0.0.1:34498)
0928 02:22:22.273567 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:22.273670 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:43991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260928 02:22:22.334833 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:22.334216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:43991 (local address 127.0.0.1:34514)
0928 02:22:22.334491 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:22.334615 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:43991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 02:22:22.336151 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:22.335665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:43991 (local address 127.0.0.1:34518)
0928 02:22:22.335856 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:22.336007 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:43991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 02:22:22.359695 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 22530
I20260928 02:22:22.377110 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 22660
I20260928 02:22:22.393858 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:22.377947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55160 (local address 127.4.140.195:33695)
0928 02:22:22.378132 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:22.378140 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:22.378166 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:22:22.378388 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:22.378393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:22.378551 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:22.378652 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:22.379063 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:22.379865 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:22.381186 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:22.381625 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:22.381755 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:22.392197 (+ 10442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:22.392208 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:22.392223 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:22.392964 (+ 741us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:22.393019 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:22.393049 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:22.393195 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:22.393361 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 02:22:22.393497 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:22:22.394560 22782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:22.417167 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:22.416468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:43991 (local address 127.0.0.1:34532)
0928 02:22:22.416780 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:22.416927 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:43991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 02:22:22.418453 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:22.417933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:43991 (local address 127.0.0.1:34548)
0928 02:22:22.418185 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:22.418280 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:43991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260928 02:22:22.543452 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 22660
I20260928 02:22:22.564093 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 22309
I20260928 02:22:22.565393 22370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:22.679946 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:22.679331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:43991 (local address 127.0.0.1:34560)
0928 02:22:22.679588 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:22.679680 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:43991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 02:22:22.680194 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 22309
2026-09-28T02:22:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3056 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:22:22 dist-test-slave-h37r krb5kdc[22838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:22:22 dist-test-slave-h37r krb5kdc[22838](info): set up 2 sockets
Sep 28 02:22:22 dist-test-slave-h37r krb5kdc[22838](info): commencing operation
krb5kdc: starting...
W20260928 02:22:23.808179 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:22:23 dist-test-slave-h37r krb5kdc[22838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:22:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:22:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:22:24.068619 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:46171
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:46239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:46171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:24.401175 22851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:24.401630 22851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:24.401800 22851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:24.409292 22851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:22:24.409457 22851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:24.409562 22851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:24.409667 22851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:22:24.409752 22851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:22:24.420008 22851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:46239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:46171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:46171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:24.421496 22851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:24.422859 22851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:24.435202 22857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:24.435338 22859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:24.435237 22856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:24.436547 22851 server_base.cc:1034] running on GCE node
I20260928 02:22:24.437443 22851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:24.439040 22851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:24.440289 22851 hybrid_clock.cc:648] HybridClock initialized: now 1790562144440241 us; error 59 us; skew 500 ppm
I20260928 02:22:24.440677 22851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:24 dist-test-slave-h37r krb5kdc[22838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562144, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:24.444087 22851 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:22:24.445751 22851 webserver.cc:460] Webserver started at http://127.4.140.254:46107/ using document root <none> and password file <none>
I20260928 02:22:24.446498 22851 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:24.446651 22851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:24.446975 22851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:24.450124 22851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "16b0480e78e24a1b9b12b94d4b932777"
format_stamp: "Formatted at 2026-09-28 02:22:24 on dist-test-slave-h37r"
server_key: "19aaca51d927583e626ce2494303bcfb"
server_key_iv: "30629e4600d27b54bfd2912dd901fd8c"
server_key_version: "encryptionkey@0"
I20260928 02:22:24.450898 22851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "16b0480e78e24a1b9b12b94d4b932777"
format_stamp: "Formatted at 2026-09-28 02:22:24 on dist-test-slave-h37r"
server_key: "19aaca51d927583e626ce2494303bcfb"
server_key_iv: "30629e4600d27b54bfd2912dd901fd8c"
server_key_version: "encryptionkey@0"
I20260928 02:22:24.456449 22851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:22:24.460078 22866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:24.461498 22851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:22:24.461699 22851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "16b0480e78e24a1b9b12b94d4b932777"
format_stamp: "Formatted at 2026-09-28 02:22:24 on dist-test-slave-h37r"
server_key: "19aaca51d927583e626ce2494303bcfb"
server_key_iv: "30629e4600d27b54bfd2912dd901fd8c"
server_key_version: "encryptionkey@0"
I20260928 02:22:24.461902 22851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:24.482995 22851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:24.487320 22851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:24.487615 22851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:24.507061 22851 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:46171
I20260928 02:22:24.507095 22917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:46171 every 8 connection(s)
I20260928 02:22:24.508507 22851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:22:24.512454 22918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:24.513742 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 22851
I20260928 02:22:24.513935 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:22:24.514511 4659 external_mini_cluster.cc:1428] Setting key 3380e07bf30d72144846c863692996d1
I20260928 02:22:24.527101 22918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777: Bootstrap starting.
Sep 28 02:22:24 dist-test-slave-h37r krb5kdc[22838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:22:24.530906 22918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:24.532070 22918 log.cc:824] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:24.535032 22918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777: No bootstrap required, opened a new log
I20260928 02:22:24.538618 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:24.516483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55236 (local address 127.4.140.254:46171)
0928 02:22:24.517279 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:24.517290 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:24.517333 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:22:24.518966 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:24.518998 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:24.519400 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:24.519754 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:24.521657 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:24.522985 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:24.524930 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:24.525622 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:24.525781 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:24.530760 (+ 4979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:24.530768 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:24.530784 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:24.530826 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:24.534192 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:24.534933 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:24.534939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:24.534946 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:24.535183 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:24.535597 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:24.535602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:24.535607 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:24.536223 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:24.536550 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:24.537091 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 02:22:24.537567 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":377,"threads_started":1}
I20260928 02:22:24.543180 22918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b0480e78e24a1b9b12b94d4b932777" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 46171 } }
I20260928 02:22:24.543560 22918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:24.543676 22918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b0480e78e24a1b9b12b94d4b932777, State: Initialized, Role: FOLLOWER
I20260928 02:22:24.544308 22918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b0480e78e24a1b9b12b94d4b932777" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 46171 } }
I20260928 02:22:24.544564 22918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:22:24.544703 22918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:22:24.544901 22918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:24.547541 22918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b0480e78e24a1b9b12b94d4b932777" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 46171 } }
I20260928 02:22:24.548110 22918 leader_election.cc:302] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16b0480e78e24a1b9b12b94d4b932777; no voters:
I20260928 02:22:24.548688 22918 leader_election.cc:288] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:22:24.548825 22923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:24.549502 22923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777 [term 1 LEADER]: Becoming Leader. State: Replica: 16b0480e78e24a1b9b12b94d4b932777, State: Running, Role: LEADER
I20260928 02:22:24.550104 22923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b0480e78e24a1b9b12b94d4b932777" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 46171 } }
I20260928 02:22:24.550859 22918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:22:24.554248 22924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16b0480e78e24a1b9b12b94d4b932777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b0480e78e24a1b9b12b94d4b932777" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 46171 } } }
I20260928 02:22:24.554119 22925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16b0480e78e24a1b9b12b94d4b932777. Latest consensus state: current_term: 1 leader_uuid: "16b0480e78e24a1b9b12b94d4b932777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b0480e78e24a1b9b12b94d4b932777" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 46171 } } }
I20260928 02:22:24.554839 22925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:24.554857 22924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:24.556818 22931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:22:24.563410 22931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:22:24.573145 22931 catalog_manager.cc:1347] Generated new cluster ID: a0435805da5343ec89ddb951bd332784
I20260928 02:22:24.573319 22931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:22:24.586660 22931 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:22:24.588735 22931 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:22:24.599830 22931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777: Generated new TSK 0
I20260928 02:22:24.601065 22931 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
I20260928 02:22:24.664131 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:46171
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:46239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:25.040282 22944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:25.040608 22944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:25.040706 22944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:25.048810 22944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:25.048983 22944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:25.049176 22944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:22:25.060536 22944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:46239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:46171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:25.062115 22944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:25.063586 22944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:25.078336 22949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:25.078315 22950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:25.078801 22952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:25.079697 22944 server_base.cc:1034] running on GCE node
I20260928 02:22:25.080405 22944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:25.081538 22944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:25.083212 22944 hybrid_clock.cc:648] HybridClock initialized: now 1790562145082973 us; error 76 us; skew 500 ppm
I20260928 02:22:25.083585 22944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:25 dist-test-slave-h37r krb5kdc[22838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562145, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:25.086973 22944 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:22:25.088534 22944 webserver.cc:460] Webserver started at http://127.4.140.193:46299/ using document root <none> and password file <none>
I20260928 02:22:25.089241 22944 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:25.089391 22944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:25.089741 22944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:25.092883 22944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "b877b6107d2a40888709e5b51b80bfc7"
format_stamp: "Formatted at 2026-09-28 02:22:25 on dist-test-slave-h37r"
server_key: "adf7ff6e1598f20d1ee6aaae4c6befab"
server_key_iv: "321a23b7d07f822c9010dd13d1b55b7e"
server_key_version: "encryptionkey@0"
I20260928 02:22:25.093709 22944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "b877b6107d2a40888709e5b51b80bfc7"
format_stamp: "Formatted at 2026-09-28 02:22:25 on dist-test-slave-h37r"
server_key: "adf7ff6e1598f20d1ee6aaae4c6befab"
server_key_iv: "321a23b7d07f822c9010dd13d1b55b7e"
server_key_version: "encryptionkey@0"
I20260928 02:22:25.099169 22944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:22:25.103139 22959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:25.104677 22944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:22:25.104866 22944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "b877b6107d2a40888709e5b51b80bfc7"
format_stamp: "Formatted at 2026-09-28 02:22:25 on dist-test-slave-h37r"
server_key: "adf7ff6e1598f20d1ee6aaae4c6befab"
server_key_iv: "321a23b7d07f822c9010dd13d1b55b7e"
server_key_version: "encryptionkey@0"
I20260928 02:22:25.105077 22944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:25.119319 22944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:25.123847 22944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:25.124256 22944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:25.125330 22944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:22:25.126719 22944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:22:25.126830 22944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:25.126986 22944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:22:25.127063 22944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:25.162437 22944 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:40507
I20260928 02:22:25.162483 23071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:40507 every 8 connection(s)
I20260928 02:22:25.164033 22944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:22:25.166947 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 22944
I20260928 02:22:25.167155 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:22:25.167601 4659 external_mini_cluster.cc:1428] Setting key 87ddd5443fb2d82734cc80846641c581
Sep 28 02:22:25 dist-test-slave-h37r krb5kdc[22838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562145, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:22:25.186965 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:25.166823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:42059 (local address 127.4.140.254:46171)
0928 02:22:25.167350 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:25.167357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:25.168508 (+ 1151us) server_negotiation.cc:407] Connection header received
0928 02:22:25.170941 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:25.170947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:25.171033 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:25.171169 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:25.173050 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:25.173796 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:25.175493 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:25.175995 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:25.176147 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:25.180419 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:25.180426 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:25.180430 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:25.180464 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:25.183254 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:25.184178 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:25.184186 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:25.184195 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:25.184348 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:25.184859 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:25.184863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:25.184865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:25.185197 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:25.185369 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:25.186265 (+ 896us) server_negotiation.cc:299] Negotiation successful
0928 02:22:25.186543 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":264,"threads_started":1}
I20260928 02:22:25.188122 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:25.167369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:46171 (local address 127.4.140.193:42059)
0928 02:22:25.168323 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:25.168379 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:25.170637 (+ 2258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:25.171462 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:25.171471 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:25.172228 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:22:25.172869 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:25.172888 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:25.173966 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:25.173971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:25.175326 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:25.175338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:25.176245 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:25.176254 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:25.176540 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:25.177431 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:22:25.177457 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:22:25.180146 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:22:25.183432 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:22:25.183442 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:22:25.183461 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:22:25.184028 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:22:25.184470 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:22:25.184475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:22:25.184479 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:22:25.184722 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:22:25.185340 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:22:25.185349 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:22:25.185769 (+ 420us) client_negotiation.cc:769] Sending connection context
0928 02:22:25.186132 (+ 363us) client_negotiation.cc:240] Negotiation successful
0928 02:22:25.186731 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":319,"threads_started":1}
I20260928 02:22:25.190065 23072 heartbeater.cc:347] Connected to a master server at 127.4.140.254:46171
I20260928 02:22:25.190394 23072 heartbeater.cc:464] Registering TS with master...
I20260928 02:22:25.191187 23072 heartbeater.cc:511] Master 127.4.140.254:46171 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.140.194@KRBTEST.COM; defaulting to no policy
I20260928 02:22:25.193535 22883 ts_manager.cc:194] Registered new tserver with Master: b877b6107d2a40888709e5b51b80bfc7 (127.4.140.193:40507)
I20260928 02:22:25.195022 22883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:42059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
I20260928 02:22:25.223719 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:46171
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:46239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:25.547662 23078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:25.548004 23078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:25.548121 23078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:25.555349 23078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:25.555444 23078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:25.555550 23078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:22:25.565189 23078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:46239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:46171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:25.566597 23078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:25.567947 23078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:25.581346 23084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:25.581434 23086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:25.581458 23083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:25.582091 23078 server_base.cc:1034] running on GCE node
I20260928 02:22:25.582698 23078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:25.583698 23078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:25.584918 23078 hybrid_clock.cc:648] HybridClock initialized: now 1790562145584868 us; error 44 us; skew 500 ppm
I20260928 02:22:25.585230 23078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:25 dist-test-slave-h37r krb5kdc[22838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562145, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:25.588532 23078 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:22:25.589895 23078 webserver.cc:460] Webserver started at http://127.4.140.194:44691/ using document root <none> and password file <none>
I20260928 02:22:25.590512 23078 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:25.590621 23078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:25.590878 23078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:25.593787 23078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "68d3b48a560e4dc0b367b626cac2607b"
format_stamp: "Formatted at 2026-09-28 02:22:25 on dist-test-slave-h37r"
server_key: "37f1e1029c88571ab7d72ee68daca816"
server_key_iv: "3b404ce5b7e36e1ac458ae3d9993bfde"
server_key_version: "encryptionkey@0"
I20260928 02:22:25.594520 23078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "68d3b48a560e4dc0b367b626cac2607b"
format_stamp: "Formatted at 2026-09-28 02:22:25 on dist-test-slave-h37r"
server_key: "37f1e1029c88571ab7d72ee68daca816"
server_key_iv: "3b404ce5b7e36e1ac458ae3d9993bfde"
server_key_version: "encryptionkey@0"
I20260928 02:22:25.599627 23078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 02:22:25.603116 23093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:25.604507 23078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:22:25.604688 23078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "68d3b48a560e4dc0b367b626cac2607b"
format_stamp: "Formatted at 2026-09-28 02:22:25 on dist-test-slave-h37r"
server_key: "37f1e1029c88571ab7d72ee68daca816"
server_key_iv: "3b404ce5b7e36e1ac458ae3d9993bfde"
server_key_version: "encryptionkey@0"
I20260928 02:22:25.604904 23078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:25.619684 23078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:25.623669 23078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:25.623979 23078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:25.624907 23078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:22:25.626200 23078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:22:25.626305 23078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:25.626449 23078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:22:25.626523 23078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:25.660485 23078 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:39859
I20260928 02:22:25.660535 23205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:39859 every 8 connection(s)
I20260928 02:22:25.661862 23078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:22:25.667951 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 23078
I20260928 02:22:25.668140 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:22:25.668744 4659 external_mini_cluster.cc:1428] Setting key 1ddbcb28b6a27d309dfd04cca786823c
Sep 28 02:22:25 dist-test-slave-h37r krb5kdc[22838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562145, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:22:25.684612 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:25.664639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:33817 (local address 127.4.140.254:46171)
0928 02:22:25.664824 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:25.664828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:25.666261 (+ 1433us) server_negotiation.cc:407] Connection header received
0928 02:22:25.668530 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:25.668535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:25.668610 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:25.668700 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:25.670474 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:25.671495 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:25.673343 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:25.674085 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:25.674227 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:25.678580 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:25.678589 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:25.678597 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:25.678645 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:25.681558 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:25.682294 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:25.682299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:25.682301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:25.682424 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:25.682868 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:25.682872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:25.682874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:25.683133 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:25.683285 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:25.684007 (+ 722us) server_negotiation.cc:299] Negotiation successful
0928 02:22:25.684239 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:22:25.685622 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:25.665214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:46171 (local address 127.4.140.194:33817)
0928 02:22:25.666060 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:25.666131 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:25.668214 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:25.668994 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:25.669007 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:25.669666 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:22:25.670262 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:25.670281 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:25.671655 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:25.671659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:25.673190 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:25.673202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:25.674291 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:25.674296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:25.674506 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:25.675236 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:22:25.675276 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:22:25.678318 (+ 3042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:22:25.681707 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:22:25.681716 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:22:25.681740 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:22:25.682156 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:22:25.682542 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:22:25.682547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:22:25.682551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:22:25.682741 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:22:25.683269 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:22:25.683277 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:22:25.683624 (+ 347us) client_negotiation.cc:769] Sending connection context
0928 02:22:25.683884 (+ 260us) client_negotiation.cc:240] Negotiation successful
0928 02:22:25.684465 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":360,"threads_started":1}
I20260928 02:22:25.687166 23206 heartbeater.cc:347] Connected to a master server at 127.4.140.254:46171
I20260928 02:22:25.687480 23206 heartbeater.cc:464] Registering TS with master...
I20260928 02:22:25.688266 23206 heartbeater.cc:511] Master 127.4.140.254:46171 requested a full tablet report, sending...
I20260928 02:22:25.689872 22883 ts_manager.cc:194] Registered new tserver with Master: 68d3b48a560e4dc0b367b626cac2607b (127.4.140.194:39859)
I20260928 02:22:25.691210 22883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:33817
WARNING: no policy specified for kudu/127.4.140.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
I20260928 02:22:25.727865 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:46171
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:46239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:26.062148 23211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:26.062439 23211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:26.062598 23211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:26.070142 23211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:26.070262 23211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:26.070442 23211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:22:26.080664 23211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:46239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:46171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:26.082104 23211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:26.083472 23211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:26.097352 23219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:26.097364 23217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:26.097446 23216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:26.097998 23211 server_base.cc:1034] running on GCE node
I20260928 02:22:26.098662 23211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:26.099649 23211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:26.100883 23211 hybrid_clock.cc:648] HybridClock initialized: now 1790562146100825 us; error 56 us; skew 500 ppm
I20260928 02:22:26.101217 23211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:26 dist-test-slave-h37r krb5kdc[22838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562146, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:26.104700 23211 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:22:26.106153 23211 webserver.cc:460] Webserver started at http://127.4.140.195:44743/ using document root <none> and password file <none>
I20260928 02:22:26.106796 23211 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:26.106892 23211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:26.107177 23211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:26.110476 23211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "310c57a2c9bf4f1a9950da1275ee153a"
format_stamp: "Formatted at 2026-09-28 02:22:26 on dist-test-slave-h37r"
server_key: "356c9968249fffad85feb3fbeb1d4f91"
server_key_iv: "3e6cfaecc8505e912eccbd1409967a91"
server_key_version: "encryptionkey@0"
I20260928 02:22:26.111202 23211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "310c57a2c9bf4f1a9950da1275ee153a"
format_stamp: "Formatted at 2026-09-28 02:22:26 on dist-test-slave-h37r"
server_key: "356c9968249fffad85feb3fbeb1d4f91"
server_key_iv: "3e6cfaecc8505e912eccbd1409967a91"
server_key_version: "encryptionkey@0"
I20260928 02:22:26.116978 23211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:22:26.120451 23226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:26.122014 23211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 02:22:26.122215 23211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "310c57a2c9bf4f1a9950da1275ee153a"
format_stamp: "Formatted at 2026-09-28 02:22:26 on dist-test-slave-h37r"
server_key: "356c9968249fffad85feb3fbeb1d4f91"
server_key_iv: "3e6cfaecc8505e912eccbd1409967a91"
server_key_version: "encryptionkey@0"
I20260928 02:22:26.122443 23211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:26.134562 23211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:26.138412 23211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:26.138717 23211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:26.139627 23211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:22:26.140975 23211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:22:26.141075 23211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:26.141222 23211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:22:26.141296 23211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:26.174108 23211 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:33733
I20260928 02:22:26.174153 23338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:33733 every 8 connection(s)
I20260928 02:22:26.175347 23211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:22:26.183207 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 23211
I20260928 02:22:26.183379 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:22:26.183821 4659 external_mini_cluster.cc:1428] Setting key 1f46b3420eb5d587afd499d1c13765bb
Sep 28 02:22:26 dist-test-slave-h37r krb5kdc[22838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562146, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:22:26.194731 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:26.178013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:49241 (local address 127.4.140.254:46171)
0928 02:22:26.178196 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:26.178200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:26.179242 (+ 1042us) server_negotiation.cc:407] Connection header received
0928 02:22:26.181017 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:26.181022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:26.181087 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:26.181174 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:26.182560 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.183316 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.185264 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.185785 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.185919 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:26.189319 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:26.189324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:26.189327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:26.189358 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:26.191823 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:26.192553 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:26.192558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:26.192562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:26.192688 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:26.193112 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:26.193117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:26.193121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:26.193379 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:26.193538 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:26.194179 (+ 641us) server_negotiation.cc:299] Negotiation successful
0928 02:22:26.194385 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:22:26.195608 23341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:26.178437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:46171 (local address 127.4.140.195:49241)
0928 02:22:26.179086 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:26.179122 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:26.180742 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:26.181375 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:26.181383 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:26.181861 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:22:26.182407 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:26.182423 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.183486 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.183493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:26.185131 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:26.185143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.186012 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.186021 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:26.186255 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:26.187000 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:22:26.187020 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:22:26.189118 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:22:26.191983 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:22:26.191991 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:22:26.192006 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:22:26.192401 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:22:26.192798 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:22:26.192803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:22:26.192807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:22:26.192993 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:22:26.193520 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:22:26.193528 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:22:26.193820 (+ 292us) client_negotiation.cc:769] Sending connection context
0928 02:22:26.194072 (+ 252us) client_negotiation.cc:240] Negotiation successful
0928 02:22:26.194548 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":245,"threads_started":1}
I20260928 02:22:26.197084 23339 heartbeater.cc:347] Connected to a master server at 127.4.140.254:46171
I20260928 02:22:26.197436 23339 heartbeater.cc:464] Registering TS with master...
I20260928 02:22:26.198139 23072 heartbeater.cc:503] Master 127.4.140.254:46171 was elected leader, sending a full tablet report...
I20260928 02:22:26.198220 23339 heartbeater.cc:511] Master 127.4.140.254:46171 requested a full tablet report, sending...
I20260928 02:22:26.199770 22882 ts_manager.cc:194] Registered new tserver with Master: 310c57a2c9bf4f1a9950da1275ee153a (127.4.140.195:33733)
I20260928 02:22:26.200649 22882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:49241
I20260928 02:22:26.211130 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:22:26.227979 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:26.215093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46380 (local address 127.4.140.254:46171)
0928 02:22:26.215259 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:26.215263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:26.215821 (+ 558us) server_negotiation.cc:407] Connection header received
0928 02:22:26.216029 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:26.216034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:26.216094 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:26.216174 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:26.217564 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.218270 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.219529 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.219999 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.220092 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:26.222519 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:26.222524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:26.222527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:26.222558 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:26.225116 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:26.225868 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:26.225874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:26.225878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:26.226010 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:26.226476 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:26.226494 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:26.226501 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:26.226820 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:26.227107 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:26.227449 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 02:22:26.227629 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:22:26.231794 22882 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:22:26.238621 22882 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:22:26.265367 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:26.250787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:39859 (local address 127.0.0.1:40274)
0928 02:22:26.251763 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:26.251811 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:26.252035 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:26.254326 (+ 2291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:26.254337 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:26.254380 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:26.254976 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:26.254992 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.257813 (+ 2821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.257822 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:26.261211 (+ 3389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:26.261236 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.264305 (+ 3069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.264312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:26.264535 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:26.264596 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 02:22:26.264742 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 02:22:26.264955 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"threads_started":1}
I20260928 02:22:26.265544 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:26.250992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.4.140.194:39859)
0928 02:22:26.253228 (+ 2236us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:26.253241 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:26.253280 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:22:26.253412 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:26.253423 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:26.253740 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:26.253974 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:26.255252 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.257616 (+ 2364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.261516 (+ 3900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.263666 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.263849 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:26.264157 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:26.264957 (+ 800us) server_negotiation.cc:299] Negotiation successful
0928 02:22:26.265191 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2062,"thread_start_us":1924,"threads_started":1}
I20260928 02:22:26.271420 23141 tablet_service.cc:1467] Processing CreateTablet for tablet adf77430a4354ad3bd0d4ec8f3c878d4 (DEFAULT_TABLE table=test-table [id=b8e63a780ac64d169e457d3b9ea40059]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:22:26.273031 23141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf77430a4354ad3bd0d4ec8f3c878d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:26.273703 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:26.251027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:33733 (local address 127.0.0.1:47564)
0928 02:22:26.265930 (+ 14903us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:26.265954 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:26.266090 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:26.266915 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:26.266921 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:26.266953 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:26.267187 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:26.267200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.269428 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.269433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:26.271230 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:26.271245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.272803 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.272809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:26.272997 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:26.273032 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:22:26.273159 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 02:22:26.273297 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14763,"mutex_wait_us":1,"spinlock_wait_cycles":3456,"thread_start_us":291,"threads_started":1}
I20260928 02:22:26.274080 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:26.251711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47564 (local address 127.4.140.195:33733)
0928 02:22:26.264629 (+ 12918us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:26.264638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:26.266232 (+ 1594us) server_negotiation.cc:407] Connection header received
0928 02:22:26.266361 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:26.266370 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:26.266680 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:26.267363 (+ 683us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:26.267651 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.269232 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.271400 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.272635 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.272777 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:26.273024 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:26.273315 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 02:22:26.273478 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12691,"thread_start_us":12921,"threads_started":1}
I20260928 02:22:26.274838 23355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:26.253120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:40507 (local address 127.0.0.1:44906)
0928 02:22:26.266903 (+ 13783us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:26.266920 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:26.267037 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:26.267775 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:26.267780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:26.267804 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:26.268201 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:26.268216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.270216 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.270224 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:26.271763 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:26.271777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.274042 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.274049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:26.274219 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:26.274250 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:22:26.274343 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 02:22:26.274466 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13612,"spinlock_wait_cycles":1792,"thread_start_us":13435,"threads_started":1}
I20260928 02:22:26.278192 23274 tablet_service.cc:1467] Processing CreateTablet for tablet adf77430a4354ad3bd0d4ec8f3c878d4 (DEFAULT_TABLE table=test-table [id=b8e63a780ac64d169e457d3b9ea40059]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:22:26.279363 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:26.253838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44906 (local address 127.4.140.193:40507)
0928 02:22:26.266395 (+ 12557us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:26.266403 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:26.267172 (+ 769us) server_negotiation.cc:407] Connection header received
0928 02:22:26.267283 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:26.267293 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:26.267565 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:26.267775 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:26.268732 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.270083 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.271970 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.273516 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.273656 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:26.273882 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:26.275154 (+ 1272us) server_negotiation.cc:299] Negotiation successful
0928 02:22:26.279124 (+ 3970us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12317,"thread_start_us":12150,"threads_started":1}
I20260928 02:22:26.279752 23274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf77430a4354ad3bd0d4ec8f3c878d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:26.282689 23007 tablet_service.cc:1467] Processing CreateTablet for tablet adf77430a4354ad3bd0d4ec8f3c878d4 (DEFAULT_TABLE table=test-table [id=b8e63a780ac64d169e457d3b9ea40059]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:22:26.284123 23007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf77430a4354ad3bd0d4ec8f3c878d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:26.288305 23357 tablet_bootstrap.cc:492] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b: Bootstrap starting.
I20260928 02:22:26.292515 23357 tablet_bootstrap.cc:654] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:26.293745 23357 log.cc:824] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:26.293848 23358 tablet_bootstrap.cc:492] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a: Bootstrap starting.
I20260928 02:22:26.296845 23360 tablet_bootstrap.cc:492] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7: Bootstrap starting.
I20260928 02:22:26.297294 23358 tablet_bootstrap.cc:654] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:26.297758 23357 tablet_bootstrap.cc:492] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b: No bootstrap required, opened a new log
I20260928 02:22:26.298020 23357 ts_tablet_manager.cc:1397] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 02:22:26.298609 23358 log.cc:824] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:26.300215 23360 tablet_bootstrap.cc:654] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:26.301578 23360 log.cc:824] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:26.301945 23358 tablet_bootstrap.cc:492] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a: No bootstrap required, opened a new log
I20260928 02:22:26.302312 23358 ts_tablet_manager.cc:1397] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 02:22:26.306427 23360 tablet_bootstrap.cc:492] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7: No bootstrap required, opened a new log
I20260928 02:22:26.306267 23357 raft_consensus.cc:348] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3b48a560e4dc0b367b626cac2607b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39859 } } peers { permanent_uuid: "310c57a2c9bf4f1a9950da1275ee153a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33733 } } peers { permanent_uuid: "b877b6107d2a40888709e5b51b80bfc7" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 40507 } }
I20260928 02:22:26.306696 23360 ts_tablet_manager.cc:1397] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260928 02:22:26.306733 23357 raft_consensus.cc:374] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:26.306859 23357 raft_consensus.cc:729] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68d3b48a560e4dc0b367b626cac2607b, State: Initialized, Role: FOLLOWER
I20260928 02:22:26.307461 23357 consensus_queue.cc:260] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3b48a560e4dc0b367b626cac2607b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39859 } } peers { permanent_uuid: "310c57a2c9bf4f1a9950da1275ee153a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33733 } } peers { permanent_uuid: "b877b6107d2a40888709e5b51b80bfc7" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 40507 } }
I20260928 02:22:26.308719 23206 heartbeater.cc:503] Master 127.4.140.254:46171 was elected leader, sending a full tablet report...
I20260928 02:22:26.309706 23357 ts_tablet_manager.cc:1428] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260928 02:22:26.312934 23358 raft_consensus.cc:348] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3b48a560e4dc0b367b626cac2607b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39859 } } peers { permanent_uuid: "310c57a2c9bf4f1a9950da1275ee153a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33733 } } peers { permanent_uuid: "b877b6107d2a40888709e5b51b80bfc7" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 40507 } }
I20260928 02:22:26.313350 23358 raft_consensus.cc:374] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:26.313450 23358 raft_consensus.cc:729] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310c57a2c9bf4f1a9950da1275ee153a, State: Initialized, Role: FOLLOWER
I20260928 02:22:26.314049 23358 consensus_queue.cc:260] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3b48a560e4dc0b367b626cac2607b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39859 } } peers { permanent_uuid: "310c57a2c9bf4f1a9950da1275ee153a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33733 } } peers { permanent_uuid: "b877b6107d2a40888709e5b51b80bfc7" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 40507 } }
I20260928 02:22:26.316520 23339 heartbeater.cc:503] Master 127.4.140.254:46171 was elected leader, sending a full tablet report...
I20260928 02:22:26.317612 23360 raft_consensus.cc:348] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3b48a560e4dc0b367b626cac2607b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39859 } } peers { permanent_uuid: "310c57a2c9bf4f1a9950da1275ee153a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33733 } } peers { permanent_uuid: "b877b6107d2a40888709e5b51b80bfc7" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 40507 } }
I20260928 02:22:26.318086 23360 raft_consensus.cc:374] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:26.318199 23360 raft_consensus.cc:729] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b877b6107d2a40888709e5b51b80bfc7, State: Initialized, Role: FOLLOWER
I20260928 02:22:26.318485 23358 ts_tablet_manager.cc:1428] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a: Time spent starting tablet: real 0.016s user 0.009s sys 0.005s
I20260928 02:22:26.318815 23360 consensus_queue.cc:260] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3b48a560e4dc0b367b626cac2607b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39859 } } peers { permanent_uuid: "310c57a2c9bf4f1a9950da1275ee153a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33733 } } peers { permanent_uuid: "b877b6107d2a40888709e5b51b80bfc7" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 40507 } }
I20260928 02:22:26.320611 23360 ts_tablet_manager.cc:1428] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260928 02:22:26.413978 23207 tablet.cc:2369] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:22:26.416394 23073 tablet.cc:2369] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:22:26.427052 23340 tablet.cc:2369] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:22:26.489401 23365 raft_consensus.cc:482] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:22:26.489712 23365 raft_consensus.cc:504] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3b48a560e4dc0b367b626cac2607b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39859 } } peers { permanent_uuid: "310c57a2c9bf4f1a9950da1275ee153a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33733 } } peers { permanent_uuid: "b877b6107d2a40888709e5b51b80bfc7" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 40507 } }
I20260928 02:22:26.491415 23365 leader_election.cc:288] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68d3b48a560e4dc0b367b626cac2607b (127.4.140.194:39859), 310c57a2c9bf4f1a9950da1275ee153a (127.4.140.195:33733)
I20260928 02:22:26.498096 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:26.491772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:44251 (local address 127.4.140.194:39859)
0928 02:22:26.492048 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:26.492058 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:26.492798 (+ 740us) server_negotiation.cc:407] Connection header received
0928 02:22:26.492988 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:26.492992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:26.493051 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:26.493145 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:26.493867 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.494554 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.496380 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.497123 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.497306 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:26.497393 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:26.497703 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 02:22:26.497830 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:22:26.498127 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:26.492164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:43333 (local address 127.4.140.195:33733)
0928 02:22:26.492362 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:26.492371 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:26.492712 (+ 341us) server_negotiation.cc:407] Connection header received
0928 02:22:26.492922 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:26.492931 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:26.492989 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:26.493083 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:26.493727 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.494465 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.496373 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.497120 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.497303 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:26.497383 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:26.497735 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 02:22:26.497858 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:22:26.498297 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:26.491817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:39859 (local address 127.4.140.193:44251)
0928 02:22:26.492698 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:26.492717 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:26.492834 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:26.493173 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:26.493181 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:26.493205 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:26.493685 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:26.493696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.494698 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.494704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:26.496217 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:26.496229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.497403 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.497411 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:26.497543 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:26.497566 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:22:26.497675 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 02:22:26.497799 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":3,"thread_start_us":422,"threads_started":1}
I20260928 02:22:26.498327 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:26.491717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:33733 (local address 127.4.140.193:43333)
0928 02:22:26.492596 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:26.492616 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:26.492785 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:26.493098 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:26.493102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:26.493125 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:26.493582 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:26.493592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.494587 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.494595 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:26.496227 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:26.496239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.497352 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.497359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:26.497560 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:26.497598 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:22:26.497708 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:22:26.497828 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":42,"spinlock_wait_cycles":24960,"thread_start_us":692,"threads_started":1}
I20260928 02:22:26.499334 23161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf77430a4354ad3bd0d4ec8f3c878d4" candidate_uuid: "b877b6107d2a40888709e5b51b80bfc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68d3b48a560e4dc0b367b626cac2607b" is_pre_election: true
I20260928 02:22:26.499399 23294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf77430a4354ad3bd0d4ec8f3c878d4" candidate_uuid: "b877b6107d2a40888709e5b51b80bfc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "310c57a2c9bf4f1a9950da1275ee153a" is_pre_election: true
I20260928 02:22:26.499814 23161 raft_consensus.cc:2454] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b877b6107d2a40888709e5b51b80bfc7 in term 0.
I20260928 02:22:26.499838 23294 raft_consensus.cc:2454] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b877b6107d2a40888709e5b51b80bfc7 in term 0.
I20260928 02:22:26.500532 22960 leader_election.cc:302] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 310c57a2c9bf4f1a9950da1275ee153a, 68d3b48a560e4dc0b367b626cac2607b, b877b6107d2a40888709e5b51b80bfc7; no voters:
I20260928 02:22:26.500947 23365 raft_consensus.cc:2781] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:22:26.501091 23365 raft_consensus.cc:482] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:22:26.501201 23365 raft_consensus.cc:3037] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:26.503269 23365 raft_consensus.cc:504] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3b48a560e4dc0b367b626cac2607b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39859 } } peers { permanent_uuid: "310c57a2c9bf4f1a9950da1275ee153a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33733 } } peers { permanent_uuid: "b877b6107d2a40888709e5b51b80bfc7" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 40507 } }
I20260928 02:22:26.503955 23365 leader_election.cc:288] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [CANDIDATE]: Term 1 election: Requested vote from peers 68d3b48a560e4dc0b367b626cac2607b (127.4.140.194:39859), 310c57a2c9bf4f1a9950da1275ee153a (127.4.140.195:33733)
I20260928 02:22:26.504302 23161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf77430a4354ad3bd0d4ec8f3c878d4" candidate_uuid: "b877b6107d2a40888709e5b51b80bfc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68d3b48a560e4dc0b367b626cac2607b"
I20260928 02:22:26.504498 23161 raft_consensus.cc:3037] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:26.504436 23294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf77430a4354ad3bd0d4ec8f3c878d4" candidate_uuid: "b877b6107d2a40888709e5b51b80bfc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "310c57a2c9bf4f1a9950da1275ee153a"
I20260928 02:22:26.504681 23294 raft_consensus.cc:3037] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:26.506731 23161 raft_consensus.cc:2454] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b877b6107d2a40888709e5b51b80bfc7 in term 1.
I20260928 02:22:26.507094 22960 leader_election.cc:302] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68d3b48a560e4dc0b367b626cac2607b, b877b6107d2a40888709e5b51b80bfc7; no voters:
I20260928 02:22:26.507140 23294 raft_consensus.cc:2454] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b877b6107d2a40888709e5b51b80bfc7 in term 1.
I20260928 02:22:26.507380 23365 raft_consensus.cc:2781] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:26.507963 23365 raft_consensus.cc:686] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [term 1 LEADER]: Becoming Leader. State: Replica: b877b6107d2a40888709e5b51b80bfc7, State: Running, Role: LEADER
I20260928 02:22:26.508540 23365 consensus_queue.cc:237] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3b48a560e4dc0b367b626cac2607b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39859 } } peers { permanent_uuid: "310c57a2c9bf4f1a9950da1275ee153a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33733 } } peers { permanent_uuid: "b877b6107d2a40888709e5b51b80bfc7" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 40507 } }
I20260928 02:22:26.513195 22882 catalog_manager.cc:5495] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b877b6107d2a40888709e5b51b80bfc7 (127.4.140.193). New cstate: current_term: 1 leader_uuid: "b877b6107d2a40888709e5b51b80bfc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3b48a560e4dc0b367b626cac2607b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "310c57a2c9bf4f1a9950da1275ee153a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b877b6107d2a40888709e5b51b80bfc7" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 40507 } health_report { overall_health: HEALTHY } } }
I20260928 02:22:26.558260 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:26.553012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44922 (local address 127.4.140.193:40507)
0928 02:22:26.553204 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:26.553209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:26.553300 (+ 91us) server_negotiation.cc:407] Connection header received
0928 02:22:26.553480 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:26.553484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:26.553560 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:26.553698 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:22:26.554037 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.554742 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.556192 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.556593 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.556695 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:26.557241 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:22:26.557788 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:22:26.557922 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:26.558025 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 02:22:26.558063 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:22:26.566280 23294 raft_consensus.cc:1261] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a [term 1 FOLLOWER]: Refusing update from remote peer b877b6107d2a40888709e5b51b80bfc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:22:26.566475 23161 raft_consensus.cc:1261] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b [term 1 FOLLOWER]: Refusing update from remote peer b877b6107d2a40888709e5b51b80bfc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:22:26.567451 23368 consensus_queue.cc:1035] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "310c57a2c9bf4f1a9950da1275ee153a" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 33733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:22:26.567866 23365 consensus_queue.cc:1035] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68d3b48a560e4dc0b367b626cac2607b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:22:26.593797 22883 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:22:26.594077 22883 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:22:26.597858 22883 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16b0480e78e24a1b9b12b94d4b932777: Sending DeleteTablet for 3 replicas of tablet adf77430a4354ad3bd0d4ec8f3c878d4
I20260928 02:22:26.598927 23141 tablet_service.cc:1514] Processing DeleteTablet for tablet adf77430a4354ad3bd0d4ec8f3c878d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:22:26 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:40274
I20260928 02:22:26.599267 23007 tablet_service.cc:1514] Processing DeleteTablet for tablet adf77430a4354ad3bd0d4ec8f3c878d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:22:26 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:44906
I20260928 02:22:26.599655 23274 tablet_service.cc:1514] Processing DeleteTablet for tablet adf77430a4354ad3bd0d4ec8f3c878d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:22:26 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:47564
I20260928 02:22:26.599839 23380 tablet_replica.cc:330] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b: stopping tablet replica
I20260928 02:22:26.600277 23380 raft_consensus.cc:2229] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:22:26.600831 23380 raft_consensus.cc:2258] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:22:26.601687 23381 tablet_replica.cc:330] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7: stopping tablet replica
I20260928 02:22:26.602210 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 22944
I20260928 02:22:26.604404 23382 tablet_replica.cc:330] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a: stopping tablet replica
I20260928 02:22:26.604779 23382 raft_consensus.cc:2229] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:22:26.605321 23382 raft_consensus.cc:2258] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:22:26.606645 23380 ts_tablet_manager.cc:1905] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:22:26.609412 23381 raft_consensus.cc:2229] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:22:26.610177 23381 raft_consensus.cc:2258] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:22:26.610795 23382 ts_tablet_manager.cc:1905] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:22:26.614485 23380 ts_tablet_manager.cc:1918] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:22:26.614725 23380 log.cc:1196] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/adf77430a4354ad3bd0d4ec8f3c878d4
I20260928 02:22:26.615425 23380 ts_tablet_manager.cc:1939] T adf77430a4354ad3bd0d4ec8f3c878d4 P 68d3b48a560e4dc0b367b626cac2607b: Deleting consensus metadata
I20260928 02:22:26.617479 22867 catalog_manager.cc:4841] TS 68d3b48a560e4dc0b367b626cac2607b (127.4.140.194:39859): tablet adf77430a4354ad3bd0d4ec8f3c878d4 (table test-table [id=b8e63a780ac64d169e457d3b9ea40059]) successfully deleted
I20260928 02:22:26.617897 23382 ts_tablet_manager.cc:1918] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:22:26.618067 23382 log.cc:1196] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/adf77430a4354ad3bd0d4ec8f3c878d4
I20260928 02:22:26.618544 23382 ts_tablet_manager.cc:1939] T adf77430a4354ad3bd0d4ec8f3c878d4 P 310c57a2c9bf4f1a9950da1275ee153a: Deleting consensus metadata
I20260928 02:22:26.620180 22870 catalog_manager.cc:4841] TS 310c57a2c9bf4f1a9950da1275ee153a (127.4.140.195:33733): tablet adf77430a4354ad3bd0d4ec8f3c878d4 (table test-table [id=b8e63a780ac64d169e457d3b9ea40059]) successfully deleted
I20260928 02:22:26.621908 23381 ts_tablet_manager.cc:1905] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:22:26 dist-test-slave-h37r krb5kdc[22838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:22:26.629992 23381 ts_tablet_manager.cc:1918] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:22:26.630185 23381 log.cc:1196] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561835360118-4659-0/minicluster-data/ts-0/wal/wals/adf77430a4354ad3bd0d4ec8f3c878d4
I20260928 02:22:26.630793 23381 ts_tablet_manager.cc:1939] T adf77430a4354ad3bd0d4ec8f3c878d4 P b877b6107d2a40888709e5b51b80bfc7: Deleting consensus metadata
I20260928 02:22:26.632905 22868 catalog_manager.cc:4841] TS b877b6107d2a40888709e5b51b80bfc7 (127.4.140.193:40507): tablet adf77430a4354ad3bd0d4ec8f3c878d4 (table test-table [id=b8e63a780ac64d169e457d3b9ea40059]) successfully deleted
I20260928 02:22:26.636837 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:26.606088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44928 (local address 127.4.140.193:40507)
0928 02:22:26.606249 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:26.606256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:26.620577 (+ 14321us) server_negotiation.cc:407] Connection header received
0928 02:22:26.620812 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:26.620818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:26.620900 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:26.621012 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:26.622354 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.623251 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.624654 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.625150 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.625300 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:26.629903 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:26.629917 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:26.629933 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:26.629971 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:26.633721 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:26.634648 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:26.634656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:26.634658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:26.634817 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:26.635239 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:26.635243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:26.635245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:26.635599 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:26.635843 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:26.636312 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 02:22:26.636501 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:22:26.637548 23067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:26.739601 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 22944
I20260928 02:22:26.759645 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 23078
Sep 28 02:22:26 dist-test-slave-h37r krb5kdc[22838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.194@KRBTEST.COM
I20260928 02:22:26.777216 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:26.761227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.4.140.194:39859)
0928 02:22:26.761410 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:26.761416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:26.761499 (+ 83us) server_negotiation.cc:407] Connection header received
0928 02:22:26.761766 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:26.761773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:26.761863 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:26.761993 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:26.763653 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.764816 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.766562 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.766960 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.767098 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:26.771273 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:26.771289 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:26.771305 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:26.771343 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:26.774240 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:26.775052 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:26.775059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:26.775062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:26.775180 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:26.775558 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:26.775564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:26.775567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:26.776012 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:26.776314 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:26.776628 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 02:22:26.776857 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:22:26.777853 23201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:26.869086 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 23078
I20260928 02:22:26.886801 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 23211
Sep 28 02:22:26 dist-test-slave-h37r krb5kdc[22838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.195@KRBTEST.COM
I20260928 02:22:26.901901 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:26.887541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47576 (local address 127.4.140.195:33733)
0928 02:22:26.887717 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:26.887722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:26.887743 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:22:26.887957 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:26.887966 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:26.888061 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:26.888159 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:26.889605 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.890321 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.891649 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:26.892151 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:26.892248 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:26.896618 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:26.896643 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:26.896664 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:26.896720 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:26.899530 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:26.900217 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:26.900224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:26.900226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:26.900369 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:26.900701 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:26.900703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:26.900705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:26.901032 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:26.901243 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:26.901486 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 02:22:26.901650 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:22:26.902554 23334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:26.992069 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 23211
I20260928 02:22:27.009106 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 22851
I20260928 02:22:27.010241 22913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:27.120721 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 22851
2026-09-28T02:22:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4439 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T02:22:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:22:27Z Disabled control of system clock
I20260928 02:22:27.195725 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:39753
--webserver_interface=127.4.140.254
--webserver_port=0
--builtin_ntp_servers=127.4.140.212:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:39753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:22:27.526710 23391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:27.527001 23391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:27.527146 23391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:27.534467 23391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:22:27.534577 23391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:27.534652 23391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:27.534744 23391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:22:27.534808 23391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:22:27.544191 23391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:39753
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:39753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:27.545512 23391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:27.547295 23391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:27.559084 23397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:27.559192 23399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:27.559114 23396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:27.560196 23391 server_base.cc:1034] running on GCE node
I20260928 02:22:27.561018 23391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:27.562386 23391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:27.563627 23391 hybrid_clock.cc:648] HybridClock initialized: now 1790562147563580 us; error 29 us; skew 500 ppm
I20260928 02:22:27.564076 23391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:22:27.566406 23391 webserver.cc:460] Webserver started at http://127.4.140.254:41087/ using document root <none> and password file <none>
I20260928 02:22:27.567220 23391 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:27.567339 23391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:27.567683 23391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:27.570729 23391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "1f3886f6ec5e4898b80f298edbfa90c5"
format_stamp: "Formatted at 2026-09-28 02:22:27 on dist-test-slave-h37r"
server_key: "e213f8ab84769dc17be0494b50a90128"
server_key_iv: "a862207819503d46bb3cc5da45994f2b"
server_key_version: "encryptionkey@0"
I20260928 02:22:27.571569 23391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "1f3886f6ec5e4898b80f298edbfa90c5"
format_stamp: "Formatted at 2026-09-28 02:22:27 on dist-test-slave-h37r"
server_key: "e213f8ab84769dc17be0494b50a90128"
server_key_iv: "a862207819503d46bb3cc5da45994f2b"
server_key_version: "encryptionkey@0"
I20260928 02:22:27.576949 23391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:22:27.582269 23405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:27.584259 23391 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260928 02:22:27.584457 23391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "1f3886f6ec5e4898b80f298edbfa90c5"
format_stamp: "Formatted at 2026-09-28 02:22:27 on dist-test-slave-h37r"
server_key: "e213f8ab84769dc17be0494b50a90128"
server_key_iv: "a862207819503d46bb3cc5da45994f2b"
server_key_version: "encryptionkey@0"
I20260928 02:22:27.584700 23391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:27.599864 23391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:27.600808 23391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:27.601099 23391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:27.618749 23391 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:39753
I20260928 02:22:27.618772 23456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:39753 every 8 connection(s)
I20260928 02:22:27.620210 23391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:22:27.624169 23457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:27.628437 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 23391
I20260928 02:22:27.628720 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:22:27.629168 4659 external_mini_cluster.cc:1428] Setting key c839d281ae5cb7eb51ca63617a832b02
I20260928 02:22:27.636749 23457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5: Bootstrap starting.
I20260928 02:22:27.640436 23457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:27.641719 23457 log.cc:824] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:27.645048 23460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:27.631076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40822 (local address 127.4.140.254:39753)
0928 02:22:27.632155 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:27.632169 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:27.632224 (+ 55us) server_negotiation.cc:407] Connection header received
0928 02:22:27.634832 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:27.634863 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:27.635494 (+ 631us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:27.635975 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:27.636514 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:27.638497 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:27.639847 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:27.640984 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:27.641155 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:27.642223 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:27.642237 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:27.642261 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:27.642928 (+ 667us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:27.643056 (+ 128us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:27.643084 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:27.643206 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:27.643413 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 02:22:27.643727 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":312,"threads_started":1}
I20260928 02:22:27.645133 23457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5: No bootstrap required, opened a new log
I20260928 02:22:27.653272 23457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3886f6ec5e4898b80f298edbfa90c5" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 39753 } }
I20260928 02:22:27.653630 23457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:27.653695 23457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f3886f6ec5e4898b80f298edbfa90c5, State: Initialized, Role: FOLLOWER
I20260928 02:22:27.654197 23457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3886f6ec5e4898b80f298edbfa90c5" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 39753 } }
I20260928 02:22:27.654407 23457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:22:27.654501 23457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:22:27.654647 23457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:27.657186 23457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3886f6ec5e4898b80f298edbfa90c5" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 39753 } }
I20260928 02:22:27.657684 23457 leader_election.cc:302] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f3886f6ec5e4898b80f298edbfa90c5; no voters:
I20260928 02:22:27.658218 23457 leader_election.cc:288] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:22:27.658425 23462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:27.659183 23462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5 [term 1 LEADER]: Becoming Leader. State: Replica: 1f3886f6ec5e4898b80f298edbfa90c5, State: Running, Role: LEADER
I20260928 02:22:27.659706 23462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3886f6ec5e4898b80f298edbfa90c5" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 39753 } }
I20260928 02:22:27.660367 23457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:22:27.662601 23463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f3886f6ec5e4898b80f298edbfa90c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3886f6ec5e4898b80f298edbfa90c5" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 39753 } } }
I20260928 02:22:27.662940 23463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:27.663820 23469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:22:27.663682 23464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f3886f6ec5e4898b80f298edbfa90c5. Latest consensus state: current_term: 1 leader_uuid: "1f3886f6ec5e4898b80f298edbfa90c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3886f6ec5e4898b80f298edbfa90c5" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 39753 } } }
I20260928 02:22:27.664520 23464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:27.668820 23469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:22:27.681257 23469 catalog_manager.cc:1347] Generated new cluster ID: 9642b32b6036484d8f4b0a6d119d7390
I20260928 02:22:27.681427 23469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:22:27.689878 23469 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:22:27.691655 23469 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:22:27.714519 23469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5: Generated new TSK 0
I20260928 02:22:27.715725 23469 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:22:27.731359 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:39753
--builtin_ntp_servers=127.4.140.212:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:22:28.063395 23481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:28.063683 23481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:28.063769 23481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:28.070992 23481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:28.071096 23481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:28.071285 23481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:22:28.080847 23481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:39753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:28.082245 23481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:28.084267 23481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:28.097484 23489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:28.097565 23487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:28.097992 23486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:28.098454 23481 server_base.cc:1034] running on GCE node
I20260928 02:22:28.099061 23481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:28.100144 23481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:28.101393 23481 hybrid_clock.cc:648] HybridClock initialized: now 1790562148101352 us; error 34 us; skew 500 ppm
I20260928 02:22:28.101730 23481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:22:28.109308 23481 webserver.cc:460] Webserver started at http://127.4.140.193:45593/ using document root <none> and password file <none>
I20260928 02:22:28.110066 23481 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:28.110179 23481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:28.110517 23481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:28.113499 23481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "bfb741267cc542dda92252fb1cf3e2ea"
format_stamp: "Formatted at 2026-09-28 02:22:28 on dist-test-slave-h37r"
server_key: "d980eda4e8cae1bbd376f276b0018cc9"
server_key_iv: "ed37a235198ead0378e87e34e39e3554"
server_key_version: "encryptionkey@0"
I20260928 02:22:28.114276 23481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "bfb741267cc542dda92252fb1cf3e2ea"
format_stamp: "Formatted at 2026-09-28 02:22:28 on dist-test-slave-h37r"
server_key: "d980eda4e8cae1bbd376f276b0018cc9"
server_key_iv: "ed37a235198ead0378e87e34e39e3554"
server_key_version: "encryptionkey@0"
I20260928 02:22:28.119319 23481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:22:28.123314 23495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:28.124747 23481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:22:28.124922 23481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "bfb741267cc542dda92252fb1cf3e2ea"
format_stamp: "Formatted at 2026-09-28 02:22:28 on dist-test-slave-h37r"
server_key: "d980eda4e8cae1bbd376f276b0018cc9"
server_key_iv: "ed37a235198ead0378e87e34e39e3554"
server_key_version: "encryptionkey@0"
I20260928 02:22:28.125139 23481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:28.145380 23481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:28.146250 23481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:28.146557 23481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:28.147558 23481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:22:28.149096 23481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:22:28.149220 23481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:28.149343 23481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:22:28.149436 23481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:28.183313 23481 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:45075
I20260928 02:22:28.183408 23607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:45075 every 8 connection(s)
I20260928 02:22:28.184680 23481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:22:28.186090 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 23481
I20260928 02:22:28.186259 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:22:28.186769 4659 external_mini_cluster.cc:1428] Setting key f3aac78ec2e0cb91f95cd85c9a2ba6e3
I20260928 02:22:28.189826 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:39753
--builtin_ntp_servers=127.4.140.212:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:22:28.198570 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:28.187604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:49965 (local address 127.4.140.254:39753)
0928 02:22:28.188244 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:28.188250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:28.189536 (+ 1286us) server_negotiation.cc:407] Connection header received
0928 02:22:28.191391 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:28.191400 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:28.191497 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:28.191625 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:28.192812 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:28.193973 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:28.195524 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:28.196266 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:28.196410 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:28.197799 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:28.197803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:28.197807 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:28.197825 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:28.197866 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:28.197879 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:28.197950 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:28.198207 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 02:22:28.198288 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":417,"threads_started":1}
I20260928 02:22:28.199647 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:28.188257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:39753 (local address 127.4.140.193:49965)
0928 02:22:28.189360 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:28.189404 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:28.191134 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:28.191805 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:28.191813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:28.191942 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:22:28.192644 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:28.192663 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:28.194141 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:28.194145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:28.195361 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:28.195372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:28.196507 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:28.196513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:28.196721 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:28.197458 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:22:28.197460 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:22:28.197499 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:22:28.197581 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:22:28.197977 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:22:28.197989 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:22:28.197999 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:22:28.198191 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 02:22:28.198636 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":340,"threads_started":1}
I20260928 02:22:28.201165 23608 heartbeater.cc:347] Connected to a master server at 127.4.140.254:39753
I20260928 02:22:28.201501 23608 heartbeater.cc:464] Registering TS with master...
I20260928 02:22:28.202389 23608 heartbeater.cc:511] Master 127.4.140.254:39753 requested a full tablet report, sending...
I20260928 02:22:28.204578 23422 ts_manager.cc:194] Registered new tserver with Master: bfb741267cc542dda92252fb1cf3e2ea (127.4.140.193:45075)
I20260928 02:22:28.206067 23422 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.140.193:49965
W20260928 02:22:28.524408 23612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:28.524688 23612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:28.524832 23612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:28.531987 23612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:28.532085 23612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:28.532269 23612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:22:28.543148 23612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:39753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:28.545014 23612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:28.547328 23612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:28.560856 23617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:28.560938 23620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:28.561479 23618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:28.565415 23612 server_base.cc:1034] running on GCE node
I20260928 02:22:28.566058 23612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:28.567075 23612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:28.568305 23612 hybrid_clock.cc:648] HybridClock initialized: now 1790562148568249 us; error 45 us; skew 500 ppm
I20260928 02:22:28.568668 23612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:22:28.571022 23612 webserver.cc:460] Webserver started at http://127.4.140.194:38021/ using document root <none> and password file <none>
I20260928 02:22:28.571724 23612 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:28.571846 23612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:28.572193 23612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:28.575011 23612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "938b389091bf464483c50e43377f2334"
format_stamp: "Formatted at 2026-09-28 02:22:28 on dist-test-slave-h37r"
server_key: "6a632782370605e317f82c2f59290ccc"
server_key_iv: "3c89216d99e2f550bf74978f4c1d7c1a"
server_key_version: "encryptionkey@0"
I20260928 02:22:28.575832 23612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "938b389091bf464483c50e43377f2334"
format_stamp: "Formatted at 2026-09-28 02:22:28 on dist-test-slave-h37r"
server_key: "6a632782370605e317f82c2f59290ccc"
server_key_iv: "3c89216d99e2f550bf74978f4c1d7c1a"
server_key_version: "encryptionkey@0"
I20260928 02:22:28.581009 23612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 02:22:28.584427 23626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:28.585769 23612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 02:22:28.585938 23612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "938b389091bf464483c50e43377f2334"
format_stamp: "Formatted at 2026-09-28 02:22:28 on dist-test-slave-h37r"
server_key: "6a632782370605e317f82c2f59290ccc"
server_key_iv: "3c89216d99e2f550bf74978f4c1d7c1a"
server_key_version: "encryptionkey@0"
I20260928 02:22:28.586160 23612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:28.605335 23612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:28.606180 23612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:28.606503 23612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:28.607535 23612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:22:28.609207 23612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:22:28.609336 23612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:28.609469 23612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:22:28.609562 23612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:28.647935 23612 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:42307
I20260928 02:22:28.648015 23738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:42307 every 8 connection(s)
I20260928 02:22:28.649304 23612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:22:28.658663 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 23612
I20260928 02:22:28.658835 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:22:28.659307 4659 external_mini_cluster.cc:1428] Setting key 40490da81d2c2fc93dd20605730326e6
I20260928 02:22:28.661968 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:28.652058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:41071 (local address 127.4.140.254:39753)
0928 02:22:28.652241 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:28.652246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:28.653703 (+ 1457us) server_negotiation.cc:407] Connection header received
0928 02:22:28.655452 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:28.655456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:28.655536 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:28.655625 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:28.656722 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:28.657422 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:28.659030 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:28.659465 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:28.659613 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:28.661127 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:28.661132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:28.661135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:28.661152 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:28.661184 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:28.661196 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:28.661295 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:28.661534 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 02:22:28.661628 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:22:28.662396 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:39753
--builtin_ntp_servers=127.4.140.212:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:22:28.663131 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:28.652626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:39753 (local address 127.4.140.194:41071)
0928 02:22:28.653532 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:28.653573 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:28.655207 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:28.655835 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:28.655844 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:28.656002 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:22:28.656575 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:28.656591 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:28.657552 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:28.657555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:28.658850 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:28.658862 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:28.659704 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:28.659712 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:28.660011 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:28.660805 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:22:28.660807 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:22:28.660846 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:22:28.660932 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:22:28.661294 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:22:28.661306 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:22:28.661316 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:22:28.661511 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 02:22:28.661979 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":287,"threads_started":1}
I20260928 02:22:28.664590 23739 heartbeater.cc:347] Connected to a master server at 127.4.140.254:39753
I20260928 02:22:28.664903 23739 heartbeater.cc:464] Registering TS with master...
I20260928 02:22:28.665658 23739 heartbeater.cc:511] Master 127.4.140.254:39753 requested a full tablet report, sending...
I20260928 02:22:28.667120 23422 ts_manager.cc:194] Registered new tserver with Master: 938b389091bf464483c50e43377f2334 (127.4.140.194:42307)
I20260928 02:22:28.668366 23422 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.140.194:41071
W20260928 02:22:28.991501 23742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:28.991787 23742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:28.991874 23742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:28.999254 23742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:28.999358 23742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:28.999469 23742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:22:29.008759 23742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:39753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:29.010155 23742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:29.011921 23742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:29.025171 23747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:29.025097 23748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:29.025133 23750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:29.025955 23742 server_base.cc:1034] running on GCE node
I20260928 02:22:29.026614 23742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:29.027627 23742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:29.028856 23742 hybrid_clock.cc:648] HybridClock initialized: now 1790562149028773 us; error 77 us; skew 500 ppm
I20260928 02:22:29.029263 23742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:22:29.031659 23742 webserver.cc:460] Webserver started at http://127.4.140.195:33671/ using document root <none> and password file <none>
I20260928 02:22:29.032452 23742 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:29.032591 23742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:29.032917 23742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:29.035810 23742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "5aa6a203de7147f28913f93edfa3bc62"
format_stamp: "Formatted at 2026-09-28 02:22:29 on dist-test-slave-h37r"
server_key: "153cab7fa6eb76ad6cd12d2767d687db"
server_key_iv: "67bdb1451f189e28e04d62e9d26cfbb3"
server_key_version: "encryptionkey@0"
I20260928 02:22:29.036674 23742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "5aa6a203de7147f28913f93edfa3bc62"
format_stamp: "Formatted at 2026-09-28 02:22:29 on dist-test-slave-h37r"
server_key: "153cab7fa6eb76ad6cd12d2767d687db"
server_key_iv: "67bdb1451f189e28e04d62e9d26cfbb3"
server_key_version: "encryptionkey@0"
I20260928 02:22:29.041682 23742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:22:29.045185 23756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:29.046597 23742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:22:29.046774 23742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "5aa6a203de7147f28913f93edfa3bc62"
format_stamp: "Formatted at 2026-09-28 02:22:29 on dist-test-slave-h37r"
server_key: "153cab7fa6eb76ad6cd12d2767d687db"
server_key_iv: "67bdb1451f189e28e04d62e9d26cfbb3"
server_key_version: "encryptionkey@0"
I20260928 02:22:29.047014 23742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:29.064842 23742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:29.065697 23742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:29.066008 23742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:29.066979 23742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:22:29.068512 23742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:22:29.068638 23742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:29.068774 23742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:22:29.068864 23742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:29.102030 23742 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:42049
I20260928 02:22:29.102083 23868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:42049 every 8 connection(s)
I20260928 02:22:29.103353 23742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:22:29.107432 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 23742
I20260928 02:22:29.107633 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:22:29.108178 4659 external_mini_cluster.cc:1428] Setting key 3f1681558cc15c8746fb070d4dfcadf1
I20260928 02:22:29.116641 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.106274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:59433 (local address 127.4.140.254:39753)
0928 02:22:29.106502 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:29.106508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:29.108264 (+ 1756us) server_negotiation.cc:407] Connection header received
0928 02:22:29.110512 (+ 2248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:29.110517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:29.110583 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:29.110656 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:29.111723 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.112619 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.114129 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.114540 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.114636 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:29.115987 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:29.115992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:29.115995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:29.116025 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:29.116055 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:29.116068 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:29.116138 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:29.116331 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 02:22:29.116402 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:22:29.117588 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.107000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:39753 (local address 127.4.140.195:59433)
0928 02:22:29.108112 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:29.108157 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:29.110186 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:29.110935 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:29.110944 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:29.111037 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:22:29.111583 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:29.111598 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.112768 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.112772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:29.113995 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:29.114007 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.114718 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.114723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:29.114921 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:29.115648 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:22:29.115651 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:22:29.115689 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:22:29.115764 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:22:29.116153 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:22:29.116161 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:22:29.116167 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:22:29.116306 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 02:22:29.116594 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":334,"threads_started":1}
I20260928 02:22:29.118878 23869 heartbeater.cc:347] Connected to a master server at 127.4.140.254:39753
I20260928 02:22:29.119170 23869 heartbeater.cc:464] Registering TS with master...
I20260928 02:22:29.119879 23869 heartbeater.cc:511] Master 127.4.140.254:39753 requested a full tablet report, sending...
I20260928 02:22:29.121273 23422 ts_manager.cc:194] Registered new tserver with Master: 5aa6a203de7147f28913f93edfa3bc62 (127.4.140.195:42049)
I20260928 02:22:29.122143 23422 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.140.195:59433
I20260928 02:22:29.123759 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:22:29.133893 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.127589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.4.140.254:39753)
0928 02:22:29.127740 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:29.127743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:29.128172 (+ 429us) server_negotiation.cc:407] Connection header received
0928 02:22:29.128363 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:29.128367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:29.128476 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:29.128571 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:29.129054 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.129768 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.131210 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.131714 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.131839 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:29.133116 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:29.133123 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:29.133127 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:29.133156 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:29.133190 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:29.133204 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:29.133396 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:29.133633 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 02:22:29.133678 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:22:29.137583 23422 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:22:29.144459 23422 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:22:29.170456 23881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.156483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:45075 (local address 127.0.0.1:57900)
0928 02:22:29.158401 (+ 1918us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:29.158442 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:29.158643 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:29.159722 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:29.159756 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:29.159792 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:29.160327 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:29.160346 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.162791 (+ 2445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.162800 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:29.166018 (+ 3218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:29.166040 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.169123 (+ 3083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.169134 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:29.169439 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:29.169526 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 02:22:29.169696 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 02:22:29.169975 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1732,"thread_start_us":1607,"threads_started":1}
I20260928 02:22:29.171486 23882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.156716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57900 (local address 127.4.140.193:45075)
0928 02:22:29.157969 (+ 1253us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:29.157976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:29.158567 (+ 591us) server_negotiation.cc:407] Connection header received
0928 02:22:29.158932 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:29.158945 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:29.159233 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:29.159435 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:29.160563 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.162591 (+ 2028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.166292 (+ 3701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.168404 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.168596 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:29.168948 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:29.170790 (+ 1842us) server_negotiation.cc:299] Negotiation successful
0928 02:22:29.171045 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1103,"thread_start_us":976,"threads_started":1}
I20260928 02:22:29.175699 23543 tablet_service.cc:1467] Processing CreateTablet for tablet 130483d928ec446f87b8c57dc959d5b2 (DEFAULT_TABLE table=test-table [id=48f7f878b51c43eabe5d9e6f32875691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:22:29.177299 23543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 130483d928ec446f87b8c57dc959d5b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:29.178289 23883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.158749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:42049 (local address 127.0.0.1:56566)
0928 02:22:29.159271 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:29.159291 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:29.159422 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:29.172289 (+ 12867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:29.172294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:29.172318 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:29.172656 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:29.172669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.174324 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.174330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:29.175861 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:29.175873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.177501 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.177508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:29.177684 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:29.177719 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:22:29.177782 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:22:29.177914 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":1,"thread_start_us":273,"threads_started":1}
I20260928 02:22:29.179530 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.159113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56566 (local address 127.4.140.195:42049)
0928 02:22:29.171681 (+ 12568us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:29.171692 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:29.171727 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:22:29.171824 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:29.171832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:29.172115 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:29.172303 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:29.172850 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.174172 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.176112 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.177291 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.178633 (+ 1342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:29.178929 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:29.179004 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 02:22:29.179173 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12396,"thread_start_us":12229,"threads_started":1}
I20260928 02:22:29.179649 23885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.159677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57672 (local address 127.4.140.194:42307)
0928 02:22:29.170497 (+ 10820us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:29.170508 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:29.170541 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:22:29.170655 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:29.170664 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:29.170941 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:29.171147 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:29.172030 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.174581 (+ 2551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.177060 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.178605 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.178737 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:29.178925 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:29.179170 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 02:22:29.179326 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10663,"thread_start_us":10569,"threads_started":1}
I20260928 02:22:29.180145 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.160025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:42307 (local address 127.0.0.1:57672)
0928 02:22:29.169796 (+ 9771us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:29.169820 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:29.169977 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:29.171401 (+ 1424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:29.171408 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:29.171435 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:29.171814 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:29.171826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.174718 (+ 2892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.174727 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:29.176830 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:29.176846 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.178795 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.178801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:29.178966 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:29.179007 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 02:22:29.179095 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 02:22:29.179230 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9597,"mutex_wait_us":1,"thread_start_us":8050,"threads_started":1}
I20260928 02:22:29.183857 23674 tablet_service.cc:1467] Processing CreateTablet for tablet 130483d928ec446f87b8c57dc959d5b2 (DEFAULT_TABLE table=test-table [id=48f7f878b51c43eabe5d9e6f32875691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:22:29.183810 23804 tablet_service.cc:1467] Processing CreateTablet for tablet 130483d928ec446f87b8c57dc959d5b2 (DEFAULT_TABLE table=test-table [id=48f7f878b51c43eabe5d9e6f32875691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:22:29.185283 23804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 130483d928ec446f87b8c57dc959d5b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:29.185359 23674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 130483d928ec446f87b8c57dc959d5b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:29.194700 23887 tablet_bootstrap.cc:492] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea: Bootstrap starting.
I20260928 02:22:29.198736 23887 tablet_bootstrap.cc:654] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:29.200163 23887 log.cc:824] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:29.200789 23888 tablet_bootstrap.cc:492] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62: Bootstrap starting.
I20260928 02:22:29.204926 23890 tablet_bootstrap.cc:492] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334: Bootstrap starting.
I20260928 02:22:29.205384 23888 tablet_bootstrap.cc:654] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:29.206578 23888 log.cc:824] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:29.206825 23887 tablet_bootstrap.cc:492] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea: No bootstrap required, opened a new log
I20260928 02:22:29.207120 23887 ts_tablet_manager.cc:1397] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260928 02:22:29.208478 23890 tablet_bootstrap.cc:654] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:29.209641 23608 heartbeater.cc:503] Master 127.4.140.254:39753 was elected leader, sending a full tablet report...
I20260928 02:22:29.209717 23890 log.cc:824] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:29.211117 23888 tablet_bootstrap.cc:492] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62: No bootstrap required, opened a new log
I20260928 02:22:29.213447 23888 ts_tablet_manager.cc:1397] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260928 02:22:29.216550 23890 tablet_bootstrap.cc:492] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334: No bootstrap required, opened a new log
I20260928 02:22:29.216828 23890 ts_tablet_manager.cc:1397] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260928 02:22:29.219084 23887 raft_consensus.cc:348] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb741267cc542dda92252fb1cf3e2ea" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45075 } } peers { permanent_uuid: "5aa6a203de7147f28913f93edfa3bc62" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 42049 } } peers { permanent_uuid: "938b389091bf464483c50e43377f2334" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42307 } }
I20260928 02:22:29.219525 23887 raft_consensus.cc:374] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:29.219689 23887 raft_consensus.cc:729] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb741267cc542dda92252fb1cf3e2ea, State: Initialized, Role: FOLLOWER
I20260928 02:22:29.220474 23887 consensus_queue.cc:260] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb741267cc542dda92252fb1cf3e2ea" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45075 } } peers { permanent_uuid: "5aa6a203de7147f28913f93edfa3bc62" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 42049 } } peers { permanent_uuid: "938b389091bf464483c50e43377f2334" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42307 } }
I20260928 02:22:29.222234 23887 ts_tablet_manager.cc:1428] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260928 02:22:29.223696 23888 raft_consensus.cc:348] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb741267cc542dda92252fb1cf3e2ea" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45075 } } peers { permanent_uuid: "5aa6a203de7147f28913f93edfa3bc62" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 42049 } } peers { permanent_uuid: "938b389091bf464483c50e43377f2334" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42307 } }
I20260928 02:22:29.224924 23888 raft_consensus.cc:374] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:29.225049 23888 raft_consensus.cc:729] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa6a203de7147f28913f93edfa3bc62, State: Initialized, Role: FOLLOWER
I20260928 02:22:29.225708 23888 consensus_queue.cc:260] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb741267cc542dda92252fb1cf3e2ea" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45075 } } peers { permanent_uuid: "5aa6a203de7147f28913f93edfa3bc62" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 42049 } } peers { permanent_uuid: "938b389091bf464483c50e43377f2334" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42307 } }
I20260928 02:22:29.226244 23890 raft_consensus.cc:348] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb741267cc542dda92252fb1cf3e2ea" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45075 } } peers { permanent_uuid: "5aa6a203de7147f28913f93edfa3bc62" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 42049 } } peers { permanent_uuid: "938b389091bf464483c50e43377f2334" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42307 } }
I20260928 02:22:29.226624 23890 raft_consensus.cc:374] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:29.226742 23890 raft_consensus.cc:729] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 938b389091bf464483c50e43377f2334, State: Initialized, Role: FOLLOWER
I20260928 02:22:29.226754 23869 heartbeater.cc:503] Master 127.4.140.254:39753 was elected leader, sending a full tablet report...
I20260928 02:22:29.227391 23888 ts_tablet_manager.cc:1428] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62: Time spent starting tablet: real 0.014s user 0.002s sys 0.009s
I20260928 02:22:29.227296 23890 consensus_queue.cc:260] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb741267cc542dda92252fb1cf3e2ea" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45075 } } peers { permanent_uuid: "5aa6a203de7147f28913f93edfa3bc62" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 42049 } } peers { permanent_uuid: "938b389091bf464483c50e43377f2334" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42307 } }
I20260928 02:22:29.228405 23739 heartbeater.cc:503] Master 127.4.140.254:39753 was elected leader, sending a full tablet report...
I20260928 02:22:29.229003 23890 ts_tablet_manager.cc:1428] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260928 02:22:29.286829 23893 raft_consensus.cc:482] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:22:29.287122 23893 raft_consensus.cc:504] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb741267cc542dda92252fb1cf3e2ea" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45075 } } peers { permanent_uuid: "5aa6a203de7147f28913f93edfa3bc62" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 42049 } } peers { permanent_uuid: "938b389091bf464483c50e43377f2334" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42307 } }
I20260928 02:22:29.288947 23893 leader_election.cc:288] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5aa6a203de7147f28913f93edfa3bc62 (127.4.140.195:42049), 938b389091bf464483c50e43377f2334 (127.4.140.194:42307)
I20260928 02:22:29.296334 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.289403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:45929 (local address 127.4.140.195:42049)
0928 02:22:29.289629 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:29.289637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:29.289989 (+ 352us) server_negotiation.cc:407] Connection header received
0928 02:22:29.290226 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:29.290235 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:29.290331 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:29.290445 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:29.291198 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.292186 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.294382 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.295321 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.295452 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:29.295499 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:29.295941 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 02:22:29.296075 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:22:29.296356 23885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.290152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:35955 (local address 127.4.140.194:42307)
0928 02:22:29.290418 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:29.290424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:29.290853 (+ 429us) server_negotiation.cc:407] Connection header received
0928 02:22:29.290972 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:29.290979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:29.291081 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:29.291170 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:29.291835 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.292813 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.294590 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.295480 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.295638 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:29.295707 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:29.295981 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 02:22:29.296109 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 02:22:29.296484 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.289200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:42049 (local address 127.4.140.193:45929)
0928 02:22:29.289878 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:29.289901 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:29.290066 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:29.290588 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:29.290592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:29.290613 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:29.291021 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:29.291034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.292321 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.292327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:29.294170 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:29.294185 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.295503 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.295512 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:29.295717 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:29.295759 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:22:29.295841 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 02:22:29.296011 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":21,"thread_start_us":374,"threads_started":1}
I20260928 02:22:29.296542 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.290012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:42307 (local address 127.4.140.193:35955)
0928 02:22:29.290760 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:29.290776 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:29.290872 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:29.291198 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:29.291202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:29.291224 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:29.291689 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:29.291701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.292950 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.292954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:29.294403 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:29.294425 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.295660 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.295665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:29.295784 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:29.295806 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:22:29.295956 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 02:22:29.296075 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":4,"thread_start_us":359,"threads_started":1}
I20260928 02:22:29.297443 23824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130483d928ec446f87b8c57dc959d5b2" candidate_uuid: "bfb741267cc542dda92252fb1cf3e2ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa6a203de7147f28913f93edfa3bc62" is_pre_election: true
I20260928 02:22:29.297443 23694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130483d928ec446f87b8c57dc959d5b2" candidate_uuid: "bfb741267cc542dda92252fb1cf3e2ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "938b389091bf464483c50e43377f2334" is_pre_election: true
I20260928 02:22:29.297955 23824 raft_consensus.cc:2454] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb741267cc542dda92252fb1cf3e2ea in term 0.
I20260928 02:22:29.297956 23694 raft_consensus.cc:2454] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb741267cc542dda92252fb1cf3e2ea in term 0.
I20260928 02:22:29.298590 23497 leader_election.cc:302] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 938b389091bf464483c50e43377f2334, bfb741267cc542dda92252fb1cf3e2ea; no voters:
I20260928 02:22:29.298977 23893 raft_consensus.cc:2781] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:22:29.299129 23893 raft_consensus.cc:482] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:22:29.299250 23893 raft_consensus.cc:3037] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:29.301481 23893 raft_consensus.cc:504] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb741267cc542dda92252fb1cf3e2ea" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45075 } } peers { permanent_uuid: "5aa6a203de7147f28913f93edfa3bc62" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 42049 } } peers { permanent_uuid: "938b389091bf464483c50e43377f2334" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42307 } }
I20260928 02:22:29.302070 23893 leader_election.cc:288] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [CANDIDATE]: Term 1 election: Requested vote from peers 5aa6a203de7147f28913f93edfa3bc62 (127.4.140.195:42049), 938b389091bf464483c50e43377f2334 (127.4.140.194:42307)
I20260928 02:22:29.302609 23824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130483d928ec446f87b8c57dc959d5b2" candidate_uuid: "bfb741267cc542dda92252fb1cf3e2ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa6a203de7147f28913f93edfa3bc62"
I20260928 02:22:29.302726 23694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130483d928ec446f87b8c57dc959d5b2" candidate_uuid: "bfb741267cc542dda92252fb1cf3e2ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "938b389091bf464483c50e43377f2334"
I20260928 02:22:29.302812 23824 raft_consensus.cc:3037] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:29.302955 23694 raft_consensus.cc:3037] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:29.305115 23824 raft_consensus.cc:2454] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb741267cc542dda92252fb1cf3e2ea in term 1.
I20260928 02:22:29.305328 23694 raft_consensus.cc:2454] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb741267cc542dda92252fb1cf3e2ea in term 1.
I20260928 02:22:29.305567 23497 leader_election.cc:302] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa6a203de7147f28913f93edfa3bc62, bfb741267cc542dda92252fb1cf3e2ea; no voters:
I20260928 02:22:29.305917 23893 raft_consensus.cc:2781] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:29.306416 23893 raft_consensus.cc:686] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [term 1 LEADER]: Becoming Leader. State: Replica: bfb741267cc542dda92252fb1cf3e2ea, State: Running, Role: LEADER
I20260928 02:22:29.306926 23893 consensus_queue.cc:237] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb741267cc542dda92252fb1cf3e2ea" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45075 } } peers { permanent_uuid: "5aa6a203de7147f28913f93edfa3bc62" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 42049 } } peers { permanent_uuid: "938b389091bf464483c50e43377f2334" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42307 } }
I20260928 02:22:29.312124 23421 catalog_manager.cc:5495] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea reported cstate change: term changed from 0 to 1, leader changed from <none> to bfb741267cc542dda92252fb1cf3e2ea (127.4.140.193). New cstate: current_term: 1 leader_uuid: "bfb741267cc542dda92252fb1cf3e2ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb741267cc542dda92252fb1cf3e2ea" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 45075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa6a203de7147f28913f93edfa3bc62" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 42049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "938b389091bf464483c50e43377f2334" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42307 } health_report { overall_health: UNKNOWN } } }
W20260928 02:22:29.355218 23870 tablet.cc:2369] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:22:29.361703 23882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.356037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.4.140.193:45075)
0928 02:22:29.356228 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:29.356237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:29.356258 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:22:29.356363 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:29.356368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:29.356496 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:29.356653 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:22:29.357043 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.357867 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.359233 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.359649 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.359778 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:29.360462 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:22:29.361033 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:22:29.361222 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:29.361339 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 02:22:29.361384 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:22:29.370769 23694 raft_consensus.cc:1261] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334 [term 1 FOLLOWER]: Refusing update from remote peer bfb741267cc542dda92252fb1cf3e2ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:22:29.370813 23824 raft_consensus.cc:1261] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62 [term 1 FOLLOWER]: Refusing update from remote peer bfb741267cc542dda92252fb1cf3e2ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:22:29.371721 23893 consensus_queue.cc:1035] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa6a203de7147f28913f93edfa3bc62" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:22:29.372126 23898 consensus_queue.cc:1035] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "938b389091bf464483c50e43377f2334" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:22:29.384138 23901 mvcc.cc:205] Tried to move back new op lower bound from 7334142563811586048 to 7334142563565998080. Current Snapshot: MvccSnapshot[applied={T|T < 7334142563811586048}]
I20260928 02:22:29.384866 23902 mvcc.cc:205] Tried to move back new op lower bound from 7334142563811586048 to 7334142563565998080. Current Snapshot: MvccSnapshot[applied={T|T < 7334142563811586048}]
I20260928 02:22:29.396783 23421 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:22:29.397075 23421 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:22:29.400703 23421 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f3886f6ec5e4898b80f298edbfa90c5: Sending DeleteTablet for 3 replicas of tablet 130483d928ec446f87b8c57dc959d5b2
I20260928 02:22:29.402326 23543 tablet_service.cc:1514] Processing DeleteTablet for tablet 130483d928ec446f87b8c57dc959d5b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:22:29 UTC) from {username='slave'} at 127.0.0.1:57900
I20260928 02:22:29.402626 23804 tablet_service.cc:1514] Processing DeleteTablet for tablet 130483d928ec446f87b8c57dc959d5b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:22:29 UTC) from {username='slave'} at 127.0.0.1:56566
I20260928 02:22:29.402964 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 23481
I20260928 02:22:29.403509 23674 tablet_service.cc:1514] Processing DeleteTablet for tablet 130483d928ec446f87b8c57dc959d5b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:22:29 UTC) from {username='slave'} at 127.0.0.1:57672
I20260928 02:22:29.415992 23914 tablet_replica.cc:330] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334: stopping tablet replica
I20260928 02:22:29.416827 23914 raft_consensus.cc:2229] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:22:29.417462 23914 raft_consensus.cc:2258] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:22:29.418375 23912 tablet_replica.cc:330] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62: stopping tablet replica
I20260928 02:22:29.418879 23912 raft_consensus.cc:2229] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:22:29.419428 23912 raft_consensus.cc:2258] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:22:29.420053 23911 tablet_replica.cc:330] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea: stopping tablet replica
I20260928 02:22:29.420653 23914 ts_tablet_manager.cc:1905] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:22:29.420766 23911 raft_consensus.cc:2229] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:22:29.421581 23911 raft_consensus.cc:2258] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:22:29.423094 23912 ts_tablet_manager.cc:1905] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:22:29.426427 23882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.403813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57916 (local address 127.4.140.193:45075)
0928 02:22:29.404020 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:29.404027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:29.415850 (+ 11823us) server_negotiation.cc:407] Connection header received
0928 02:22:29.416772 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:29.416778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:29.416887 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:29.416988 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:29.418163 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.419371 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.421463 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.422148 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.422254 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:29.423742 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:29.423753 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:29.423770 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:29.424587 (+ 817us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:29.424651 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:29.424680 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:29.424763 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:29.425885 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0928 02:22:29.426035 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:22:29.427356 23911 ts_tablet_manager.cc:1905] T 130483d928ec446f87b8c57dc959d5b2 P bfb741267cc542dda92252fb1cf3e2ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:22:29.427960 23603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:29.432780 23914 ts_tablet_manager.cc:1918] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:22:29.432960 23914 log.cc:1196] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/130483d928ec446f87b8c57dc959d5b2
I20260928 02:22:29.433441 23914 ts_tablet_manager.cc:1939] T 130483d928ec446f87b8c57dc959d5b2 P 938b389091bf464483c50e43377f2334: Deleting consensus metadata
I20260928 02:22:29.433792 23912 ts_tablet_manager.cc:1918] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:22:29.433947 23912 log.cc:1196] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/130483d928ec446f87b8c57dc959d5b2
I20260928 02:22:29.434445 23912 ts_tablet_manager.cc:1939] T 130483d928ec446f87b8c57dc959d5b2 P 5aa6a203de7147f28913f93edfa3bc62: Deleting consensus metadata
I20260928 02:22:29.435258 23407 catalog_manager.cc:4841] TS 938b389091bf464483c50e43377f2334 (127.4.140.194:42307): tablet 130483d928ec446f87b8c57dc959d5b2 (table test-table [id=48f7f878b51c43eabe5d9e6f32875691]) successfully deleted
I20260928 02:22:29.436162 23407 catalog_manager.cc:4841] TS 5aa6a203de7147f28913f93edfa3bc62 (127.4.140.195:42049): tablet 130483d928ec446f87b8c57dc959d5b2 (table test-table [id=48f7f878b51c43eabe5d9e6f32875691]) successfully deleted
I20260928 02:22:29.524977 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 23481
I20260928 02:22:29.542550 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 23612
W20260928 02:22:29.542675 23407 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.140.193:45075 (error 108)
I20260928 02:22:29.544775 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.544267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:45075 (local address 127.0.0.1:57922)
0928 02:22:29.544478 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:29.544614 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 02:22:29.545120 23407 catalog_manager.cc:4584] TS bfb741267cc542dda92252fb1cf3e2ea (127.4.140.193:45075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 130483d928ec446f87b8c57dc959d5b2: Network error: Client connection negotiation failed: client connection to 127.4.140.193:45075: connect: Connection refused (error 111)
I20260928 02:22:29.549788 23885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.543441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.4.140.194:42307)
0928 02:22:29.543622 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:29.543628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:29.543680 (+ 52us) server_negotiation.cc:407] Connection header received
0928 02:22:29.543944 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:29.543954 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:29.544081 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:29.544201 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:29.544669 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.545664 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.546937 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.547327 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.547418 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:29.548558 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:29.548566 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:29.548580 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:29.549150 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:29.549196 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:29.549215 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:29.549307 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:29.549463 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 02:22:29.549545 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:22:29.550593 23734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:29.599000 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.598505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:45075 (local address 127.0.0.1:57930)
0928 02:22:29.598728 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:29.598838 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 02:22:29.600138 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.599729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:45075 (local address 127.0.0.1:57946)
0928 02:22:29.599881 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:29.600005 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 02:22:29.641731 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 23612
I20260928 02:22:29.659641 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 23742
I20260928 02:22:29.661556 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.660923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:45075 (local address 127.0.0.1:57962)
0928 02:22:29.661169 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:29.661324 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 02:22:29.662657 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.662329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:45075 (local address 127.0.0.1:57968)
0928 02:22:29.662474 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:29.662544 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 02:22:29.667145 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.660549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56570 (local address 127.4.140.195:42049)
0928 02:22:29.660721 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:29.660725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:29.660807 (+ 82us) server_negotiation.cc:407] Connection header received
0928 02:22:29.661117 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:29.661125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:29.661263 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:29.661464 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:29.661897 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.662988 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.664436 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:29.664817 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:29.664914 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:29.665905 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:29.665913 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:29.665929 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:29.666491 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:29.666536 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:29.666554 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:29.666644 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:29.666737 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 02:22:29.666818 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:22:29.667761 23864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:29.743450 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.742922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:45075 (local address 127.0.0.1:57978)
0928 02:22:29.743182 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:29.743283 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260928 02:22:29.744652 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:29.744231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:45075 (local address 127.0.0.1:57988)
0928 02:22:29.744459 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:29.744533 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 02:22:29.756467 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 23742
I20260928 02:22:29.773479 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 23391
I20260928 02:22:29.774534 23452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:29.883528 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 23391
2026-09-28T02:22:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2762 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:22:29 dist-test-slave-h37r krb5kdc[23922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:22:29 dist-test-slave-h37r krb5kdc[23922](info): set up 2 sockets
Sep 28 02:22:29 dist-test-slave-h37r krb5kdc[23922](info): commencing operation
krb5kdc: starting...
W20260928 02:22:31.008316 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:22:31 dist-test-slave-h37r krb5kdc[23922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:22:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:22:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:22:31.264065 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:44391
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:44391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:31.597065 23935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:31.597396 23935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:31.597568 23935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:31.604944 23935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:22:31.605099 23935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:31.605216 23935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:31.605314 23935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:22:31.605401 23935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:22:31.615286 23935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:44391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:44391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:31.616765 23935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:31.618155 23935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:31.630434 23940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:31.630327 23943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:31.630350 23941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:31.631141 23935 server_base.cc:1034] running on GCE node
I20260928 02:22:31.632078 23935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:31.633560 23935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:31.634801 23935 hybrid_clock.cc:648] HybridClock initialized: now 1790562151634760 us; error 62 us; skew 500 ppm
I20260928 02:22:31.635174 23935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:31 dist-test-slave-h37r krb5kdc[23922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562151, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:31.638456 23935 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:22:31.640057 23935 webserver.cc:460] Webserver started at http://127.4.140.254:36937/ using document root <none> and password file <none>
I20260928 02:22:31.640715 23935 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:31.640811 23935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:31.641086 23935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:31.644258 23935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "5723f8d273df4f09b465e0bea7794461"
format_stamp: "Formatted at 2026-09-28 02:22:31 on dist-test-slave-h37r"
server_key: "bd32105a7240b378e111afbfcd533d97"
server_key_iv: "65544d2d18bb919ecccbb42ec9398435"
server_key_version: "encryptionkey@0"
I20260928 02:22:31.644994 23935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "5723f8d273df4f09b465e0bea7794461"
format_stamp: "Formatted at 2026-09-28 02:22:31 on dist-test-slave-h37r"
server_key: "bd32105a7240b378e111afbfcd533d97"
server_key_iv: "65544d2d18bb919ecccbb42ec9398435"
server_key_version: "encryptionkey@0"
I20260928 02:22:31.650230 23935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:22:31.653838 23950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:31.655264 23935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:22:31.655398 23935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "5723f8d273df4f09b465e0bea7794461"
format_stamp: "Formatted at 2026-09-28 02:22:31 on dist-test-slave-h37r"
server_key: "bd32105a7240b378e111afbfcd533d97"
server_key_iv: "65544d2d18bb919ecccbb42ec9398435"
server_key_version: "encryptionkey@0"
I20260928 02:22:31.655599 23935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:31.674849 23935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:31.678922 23935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:31.679157 23935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:31.698076 23935 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:44391
I20260928 02:22:31.698068 24001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:44391 every 8 connection(s)
I20260928 02:22:31.699486 23935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:22:31.703421 24002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:31.708346 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 23935
I20260928 02:22:31.708529 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:22:31.709012 4659 external_mini_cluster.cc:1428] Setting key 97183a70586a9952cb3b8595e77917bd
I20260928 02:22:31.714413 24002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461: Bootstrap starting.
I20260928 02:22:31.718127 24002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:31.719363 24002 log.cc:824] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:31.722793 24002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461: No bootstrap required, opened a new log
Sep 28 02:22:31 dist-test-slave-h37r krb5kdc[23922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:22:31.733790 24005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:31.710774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48560 (local address 127.4.140.254:44391)
0928 02:22:31.711612 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:31.711623 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:31.711666 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:22:31.713424 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:31.713446 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:31.713815 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:31.714127 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:31.715989 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:31.717317 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:31.719214 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:31.720239 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:31.720463 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:31.725015 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:31.725026 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:31.725044 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:31.725100 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:31.729149 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:31.729822 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:31.729830 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:31.729839 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:31.730040 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:31.730382 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:31.730387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:31.730390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:31.731111 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:31.731415 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:31.731768 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 02:22:31.732277 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":395,"threads_started":1}
I20260928 02:22:31.734089 24002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5723f8d273df4f09b465e0bea7794461" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 44391 } }
I20260928 02:22:31.734530 24002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:31.734649 24002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5723f8d273df4f09b465e0bea7794461, State: Initialized, Role: FOLLOWER
I20260928 02:22:31.735287 24002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5723f8d273df4f09b465e0bea7794461" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 44391 } }
I20260928 02:22:31.735567 24002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:22:31.735724 24002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:22:31.735957 24002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:31.738240 24002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5723f8d273df4f09b465e0bea7794461" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 44391 } }
I20260928 02:22:31.738754 24002 leader_election.cc:302] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5723f8d273df4f09b465e0bea7794461; no voters:
I20260928 02:22:31.739404 24002 leader_election.cc:288] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:22:31.739560 24007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:31.740312 24007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461 [term 1 LEADER]: Becoming Leader. State: Replica: 5723f8d273df4f09b465e0bea7794461, State: Running, Role: LEADER
I20260928 02:22:31.740995 24007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5723f8d273df4f09b465e0bea7794461" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 44391 } }
I20260928 02:22:31.741578 24002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:22:31.744724 24009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5723f8d273df4f09b465e0bea7794461. Latest consensus state: current_term: 1 leader_uuid: "5723f8d273df4f09b465e0bea7794461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5723f8d273df4f09b465e0bea7794461" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 44391 } } }
I20260928 02:22:31.745151 24009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:31.744424 24008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5723f8d273df4f09b465e0bea7794461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5723f8d273df4f09b465e0bea7794461" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 44391 } } }
I20260928 02:22:31.745923 24008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:31.749058 24016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:22:31.753782 24016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:22:31.762702 24016 catalog_manager.cc:1347] Generated new cluster ID: d6e3acebdcf64be99541104e92c04329
I20260928 02:22:31.762812 24016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:22:31.785941 24016 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:22:31.787842 24016 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:22:31.794576 24016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461: Generated new TSK 0
I20260928 02:22:31.795457 24016 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.140.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
Entry for principal kudu/127.4.140.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab.
I20260928 02:22:31.863508 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:44391
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:32.194183 24029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:32.194468 24029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:32.194561 24029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:32.201843 24029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:32.201948 24029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:32.202152 24029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:22:32.212280 24029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.193.keytab
--principal=kudu/127.4.140.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:44391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:32.213678 24029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:32.215008 24029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:32.228510 24037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:32.228570 24034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:32.228574 24035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:32.229404 24029 server_base.cc:1034] running on GCE node
I20260928 02:22:32.229964 24029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:32.230891 24029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:32.232076 24029 hybrid_clock.cc:648] HybridClock initialized: now 1790562152232038 us; error 28 us; skew 500 ppm
I20260928 02:22:32.232383 24029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:32 dist-test-slave-h37r krb5kdc[23922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562152, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:32.235450 24029 init.cc:377] Logged in from keytab as kudu/127.4.140.193@KRBTEST.COM (short username kudu)
I20260928 02:22:32.236954 24029 webserver.cc:460] Webserver started at http://127.4.140.193:45269/ using document root <none> and password file <none>
I20260928 02:22:32.237581 24029 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:32.237677 24029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:32.237932 24029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:32.240816 24029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "5da83fcdba3b45eab860d5f3a210e436"
format_stamp: "Formatted at 2026-09-28 02:22:32 on dist-test-slave-h37r"
server_key: "65226a79c60ff96bb0455027b33deb7b"
server_key_iv: "78d606fefae9ac39a8b6448ad6aa97e3"
server_key_version: "encryptionkey@0"
I20260928 02:22:32.241619 24029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "5da83fcdba3b45eab860d5f3a210e436"
format_stamp: "Formatted at 2026-09-28 02:22:32 on dist-test-slave-h37r"
server_key: "65226a79c60ff96bb0455027b33deb7b"
server_key_iv: "78d606fefae9ac39a8b6448ad6aa97e3"
server_key_version: "encryptionkey@0"
I20260928 02:22:32.246963 24029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:22:32.250666 24044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:32.252121 24029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:22:32.252259 24029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "5da83fcdba3b45eab860d5f3a210e436"
format_stamp: "Formatted at 2026-09-28 02:22:32 on dist-test-slave-h37r"
server_key: "65226a79c60ff96bb0455027b33deb7b"
server_key_iv: "78d606fefae9ac39a8b6448ad6aa97e3"
server_key_version: "encryptionkey@0"
I20260928 02:22:32.252429 24029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:32.264114 24029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:32.268440 24029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:32.268697 24029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:32.269698 24029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:22:32.270955 24029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:22:32.271027 24029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:32.271108 24029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:22:32.271152 24029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:32.304435 24029 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:36643
I20260928 02:22:32.304473 24156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:36643 every 8 connection(s)
I20260928 02:22:32.305759 24029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:22:32.307637 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 24029
I20260928 02:22:32.307847 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:22:32.308445 4659 external_mini_cluster.cc:1428] Setting key 4f084053ec25d3419a6f7a0d9917c151
Sep 28 02:22:32 dist-test-slave-h37r krb5kdc[23922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562152, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.193@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:22:32.327526 24159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:32.308528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:60049 (local address 127.4.140.254:44391)
0928 02:22:32.309026 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:32.309036 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:32.310312 (+ 1276us) server_negotiation.cc:407] Connection header received
0928 02:22:32.312202 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:32.312208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:32.312308 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:32.312420 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:32.313953 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:32.314680 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:32.316364 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:32.316793 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:32.316932 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:32.321208 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:32.321214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:32.321217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:32.321251 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:32.324086 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:32.325070 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:32.325074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:32.325076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:32.325195 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:32.325696 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:32.325699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:32.325700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:32.326025 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:32.326134 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:32.327092 (+ 958us) server_negotiation.cc:299] Negotiation successful
0928 02:22:32.327263 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":264,"threads_started":1}
I20260928 02:22:32.329141 24160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:32.309206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:44391 (local address 127.4.140.193:60049)
0928 02:22:32.310133 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:32.310183 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:32.311864 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:32.312664 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:32.312672 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:32.313210 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:22:32.313784 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:32.313801 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:32.314856 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:32.314860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:32.316221 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:32.316234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:32.317049 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:32.317057 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:32.317290 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:32.317985 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:22:32.318030 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:22:32.320960 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:22:32.324225 (+ 3265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:22:32.324237 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:22:32.324257 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:22:32.324937 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:22:32.325305 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:22:32.325312 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:22:32.325319 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:22:32.325585 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:22:32.326160 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:22:32.326171 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:22:32.326675 (+ 504us) client_negotiation.cc:769] Sending connection context
0928 02:22:32.326996 (+ 321us) client_negotiation.cc:240] Negotiation successful
0928 02:22:32.327604 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":324,"threads_started":1}
I20260928 02:22:32.330816 24157 heartbeater.cc:347] Connected to a master server at 127.4.140.254:44391
I20260928 02:22:32.331195 24157 heartbeater.cc:464] Registering TS with master...
I20260928 02:22:32.331959 24157 heartbeater.cc:511] Master 127.4.140.254:44391 requested a full tablet report, sending...
I20260928 02:22:32.334184 23967 ts_manager.cc:194] Registered new tserver with Master: 5da83fcdba3b45eab860d5f3a210e436 (127.4.140.193:36643)
WARNING: no policy specified for kudu/127.4.140.194@KRBTEST.COM; defaulting to no policy
I20260928 02:22:32.335685 23967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.193@KRBTEST.COM'} at 127.4.140.193:60049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
Entry for principal kudu/127.4.140.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab.
I20260928 02:22:32.367939 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:44391
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:32.707401 24163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:32.707720 24163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:32.707882 24163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:32.715163 24163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:32.715313 24163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:32.715495 24163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:22:32.725587 24163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.194.keytab
--principal=kudu/127.4.140.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:44391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:32.727013 24163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:32.728403 24163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:32.741669 24171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:32.741688 24169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:32.741915 24168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:32.742453 24163 server_base.cc:1034] running on GCE node
I20260928 02:22:32.743100 24163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:32.744112 24163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:32.745306 24163 hybrid_clock.cc:648] HybridClock initialized: now 1790562152745251 us; error 46 us; skew 500 ppm
I20260928 02:22:32.745630 24163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:32 dist-test-slave-h37r krb5kdc[23922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562152, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:32.748791 24163 init.cc:377] Logged in from keytab as kudu/127.4.140.194@KRBTEST.COM (short username kudu)
I20260928 02:22:32.750277 24163 webserver.cc:460] Webserver started at http://127.4.140.194:38359/ using document root <none> and password file <none>
I20260928 02:22:32.750985 24163 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:32.751127 24163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:32.751447 24163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:32.754426 24163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "f0346a75cbba44f9862ccc51cc03560b"
format_stamp: "Formatted at 2026-09-28 02:22:32 on dist-test-slave-h37r"
server_key: "8a98fa913c7e079e124220096c867678"
server_key_iv: "364d28aa46745667e0e6eba65fabd63c"
server_key_version: "encryptionkey@0"
I20260928 02:22:32.755404 24163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "f0346a75cbba44f9862ccc51cc03560b"
format_stamp: "Formatted at 2026-09-28 02:22:32 on dist-test-slave-h37r"
server_key: "8a98fa913c7e079e124220096c867678"
server_key_iv: "364d28aa46745667e0e6eba65fabd63c"
server_key_version: "encryptionkey@0"
I20260928 02:22:32.762625 24163 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260928 02:22:32.766152 24178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:32.767505 24163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:22:32.767668 24163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "f0346a75cbba44f9862ccc51cc03560b"
format_stamp: "Formatted at 2026-09-28 02:22:32 on dist-test-slave-h37r"
server_key: "8a98fa913c7e079e124220096c867678"
server_key_iv: "364d28aa46745667e0e6eba65fabd63c"
server_key_version: "encryptionkey@0"
I20260928 02:22:32.767879 24163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:32.786619 24163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:32.791033 24163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:32.791337 24163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:32.792359 24163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:22:32.793723 24163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:22:32.793797 24163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:32.793929 24163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:22:32.793979 24163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:32.827625 24163 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:36161
I20260928 02:22:32.827672 24290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:36161 every 8 connection(s)
I20260928 02:22:32.829003 24163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:22:32.832532 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 24163
I20260928 02:22:32.832741 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:22:32.833305 4659 external_mini_cluster.cc:1428] Setting key a0b2d0bb16542db438680a2346ac5c52
Sep 28 02:22:32 dist-test-slave-h37r krb5kdc[23922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562152, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.194@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:22:32.849784 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:32.831525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:50669 (local address 127.4.140.254:44391)
0928 02:22:32.831979 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:32.831987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:32.833260 (+ 1273us) server_negotiation.cc:407] Connection header received
0928 02:22:32.835390 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:32.835396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:32.835492 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:32.835608 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:32.837168 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:32.837863 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:32.839404 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:32.839827 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:32.840006 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:32.843820 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:32.843825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:32.843828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:32.843861 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:32.846655 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:32.847378 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:32.847384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:32.847387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:32.847526 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:32.847987 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:32.847994 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:32.847997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:32.848281 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:32.848439 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:32.849141 (+ 702us) server_negotiation.cc:299] Negotiation successful
0928 02:22:32.849361 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":198,"threads_started":1}
I20260928 02:22:32.850797 24294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:32.832237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:44391 (local address 127.4.140.194:50669)
0928 02:22:32.833105 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:32.833145 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:32.835137 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:32.835841 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:32.835850 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:32.836437 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:22:32.837000 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:32.837019 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:32.838026 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:32.838038 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:32.839248 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:32.839260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:32.840111 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:32.840118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:32.840351 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:32.841053 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:22:32.841079 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:22:32.843582 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:22:32.846802 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:22:32.846812 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:22:32.846828 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:22:32.847245 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:22:32.847638 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:22:32.847643 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:22:32.847647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:22:32.847850 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:22:32.848410 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:22:32.848418 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:22:32.848770 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 02:22:32.849041 (+ 271us) client_negotiation.cc:240] Negotiation successful
0928 02:22:32.849599 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":346,"threads_started":1}
I20260928 02:22:32.852331 24291 heartbeater.cc:347] Connected to a master server at 127.4.140.254:44391
I20260928 02:22:32.852658 24291 heartbeater.cc:464] Registering TS with master...
I20260928 02:22:32.853377 24291 heartbeater.cc:511] Master 127.4.140.254:44391 requested a full tablet report, sending...
I20260928 02:22:32.854935 23967 ts_manager.cc:194] Registered new tserver with Master: f0346a75cbba44f9862ccc51cc03560b (127.4.140.194:36161)
I20260928 02:22:32.856194 23967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.194@KRBTEST.COM'} at 127.4.140.194:50669
WARNING: no policy specified for kudu/127.4.140.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
Entry for principal kudu/127.4.140.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab.
I20260928 02:22:32.889153 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:44391
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:33.219280 24297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:33.219584 24297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:33.219753 24297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:33.227162 24297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:33.227324 24297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:33.227495 24297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:22:33.237053 24297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.195.keytab
--principal=kudu/127.4.140.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:44391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:33.238364 24297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:33.239681 24297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:33.252902 24302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:33.252979 24305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:33.252930 24303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:33.253935 24297 server_base.cc:1034] running on GCE node
I20260928 02:22:33.254500 24297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:33.255391 24297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:33.256621 24297 hybrid_clock.cc:648] HybridClock initialized: now 1790562153256573 us; error 65 us; skew 500 ppm
I20260928 02:22:33.256944 24297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:33 dist-test-slave-h37r krb5kdc[23922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562153, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:33.268321 24297 init.cc:377] Logged in from keytab as kudu/127.4.140.195@KRBTEST.COM (short username kudu)
I20260928 02:22:33.269752 24297 webserver.cc:460] Webserver started at http://127.4.140.195:34837/ using document root <none> and password file <none>
I20260928 02:22:33.270458 24297 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:33.270603 24297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:33.270910 24297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:33.274016 24297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "40629a2a044847b3aaca6de3889000a5"
format_stamp: "Formatted at 2026-09-28 02:22:33 on dist-test-slave-h37r"
server_key: "09be213b404e202237002860f35bd2b8"
server_key_iv: "6f718ec28236647c650cb81991cc7f47"
server_key_version: "encryptionkey@0"
I20260928 02:22:33.274804 24297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "40629a2a044847b3aaca6de3889000a5"
format_stamp: "Formatted at 2026-09-28 02:22:33 on dist-test-slave-h37r"
server_key: "09be213b404e202237002860f35bd2b8"
server_key_iv: "6f718ec28236647c650cb81991cc7f47"
server_key_version: "encryptionkey@0"
I20260928 02:22:33.279989 24297 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:22:33.283461 24312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:33.284863 24297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 02:22:33.285038 24297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "40629a2a044847b3aaca6de3889000a5"
format_stamp: "Formatted at 2026-09-28 02:22:33 on dist-test-slave-h37r"
server_key: "09be213b404e202237002860f35bd2b8"
server_key_iv: "6f718ec28236647c650cb81991cc7f47"
server_key_version: "encryptionkey@0"
I20260928 02:22:33.285239 24297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:33.300287 24297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:33.304160 24297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:33.304462 24297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:33.305397 24297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:22:33.306725 24297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:22:33.306828 24297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:33.306970 24297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:22:33.307044 24297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:33.339046 24157 heartbeater.cc:503] Master 127.4.140.254:44391 was elected leader, sending a full tablet report...
I20260928 02:22:33.340358 24297 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:45175
I20260928 02:22:33.340391 24424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:45175 every 8 connection(s)
I20260928 02:22:33.341825 24297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:22:33.343545 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 24297
I20260928 02:22:33.343722 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:22:33.344229 4659 external_mini_cluster.cc:1428] Setting key 23940b116a640a081d2a024ad971f892
Sep 28 02:22:33 dist-test-slave-h37r krb5kdc[23922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562153, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.195@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:22:33.361814 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:33.344607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:55743 (local address 127.4.140.254:44391)
0928 02:22:33.344800 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:33.344806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:33.346302 (+ 1496us) server_negotiation.cc:407] Connection header received
0928 02:22:33.348354 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:33.348358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:33.348436 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:33.348528 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:33.349916 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.350656 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.352250 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.352637 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.352731 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:33.356037 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:33.356058 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:33.356065 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:33.356108 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:33.358950 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:33.359824 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:33.359829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:33.359831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:33.360007 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:33.360384 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:33.360387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:33.360389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:33.360668 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:33.360782 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:33.361424 (+ 642us) server_negotiation.cc:299] Negotiation successful
0928 02:22:33.361588 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:22:33.362782 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:33.345134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:44391 (local address 127.4.140.195:55743)
0928 02:22:33.346095 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:33.346181 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:33.348094 (+ 1913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:33.348694 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:33.348701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:33.349222 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:22:33.349771 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:33.349787 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.350786 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.350790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:33.352123 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:33.352136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.352822 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.352827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:33.353038 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:33.353643 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:22:33.353666 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:22:33.355786 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:22:33.359080 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:22:33.359086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:22:33.359099 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:22:33.359713 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:22:33.360104 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:22:33.360107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:22:33.360109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:22:33.360287 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:22:33.360775 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:22:33.360781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:22:33.361095 (+ 314us) client_negotiation.cc:769] Sending connection context
0928 02:22:33.361320 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 02:22:33.361705 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":351,"threads_started":1}
I20260928 02:22:33.364248 24425 heartbeater.cc:347] Connected to a master server at 127.4.140.254:44391
I20260928 02:22:33.364517 24425 heartbeater.cc:464] Registering TS with master...
I20260928 02:22:33.365182 24425 heartbeater.cc:511] Master 127.4.140.254:44391 requested a full tablet report, sending...
I20260928 02:22:33.366524 23967 ts_manager.cc:194] Registered new tserver with Master: 40629a2a044847b3aaca6de3889000a5 (127.4.140.195:45175)
I20260928 02:22:33.367308 23967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.140.195@KRBTEST.COM'} at 127.4.140.195:55743
I20260928 02:22:33.371570 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:22:33.389592 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:33.375625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48576 (local address 127.4.140.254:44391)
0928 02:22:33.375813 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:33.375822 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:33.376272 (+ 450us) server_negotiation.cc:407] Connection header received
0928 02:22:33.376473 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:33.376478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:33.376557 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:33.376659 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:33.378106 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.378848 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.380221 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.380632 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.380767 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:33.383869 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:33.383876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:33.383881 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:33.383948 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:33.386924 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:33.387557 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:33.387561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:33.387563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:33.387681 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:33.388100 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:33.388104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:33.388106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:33.388489 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:33.388684 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:33.389079 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 02:22:33.389276 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:22:33.393594 23967 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:22:33.400580 23967 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:22:33.426545 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:33.413244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44816 (local address 127.4.140.194:36161)
0928 02:22:33.414078 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:33.414092 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:33.414809 (+ 717us) server_negotiation.cc:407] Connection header received
0928 02:22:33.415181 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:33.415189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:33.415447 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:33.415642 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:33.416740 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.418813 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.422589 (+ 3776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.424618 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.424768 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:33.425075 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:33.425986 (+ 911us) server_negotiation.cc:299] Negotiation successful
0928 02:22:33.426198 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":315,"threads_started":1}
I20260928 02:22:33.428169 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:33.413244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:36161 (local address 127.0.0.1:44816)
0928 02:22:33.414654 (+ 1410us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:33.414717 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:33.414952 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:33.415960 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:33.415969 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:33.416002 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:33.416491 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:33.416508 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.419063 (+ 2555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.419072 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:33.422359 (+ 3287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:33.422381 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.425352 (+ 2971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.425362 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:33.425649 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:33.425715 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 02:22:33.426857 (+ 1142us) client_negotiation.cc:240] Negotiation successful
0928 02:22:33.427090 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1185,"thread_start_us":1034,"threads_started":1}
I20260928 02:22:33.432423 24226 tablet_service.cc:1467] Processing CreateTablet for tablet 2b582e9821ad42a7a0f0d6bb05b0983f (DEFAULT_TABLE table=test-table [id=11cf2bde75a5470f855ee56d1255a0e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:22:33.433863 24226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b582e9821ad42a7a0f0d6bb05b0983f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:33.435503 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:33.415822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:36643 (local address 127.0.0.1:41894)
0928 02:22:33.427716 (+ 11894us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:33.427734 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:33.427861 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:33.428696 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:33.428703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:33.428733 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:33.429118 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:33.429129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.431186 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.431189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:33.432686 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:33.432697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.434666 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.434673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:33.434849 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:33.434885 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:22:33.434980 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 02:22:33.435123 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11757,"thread_start_us":10438,"threads_started":1}
I20260928 02:22:33.436147 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:33.414181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:45175 (local address 127.0.0.1:39112)
0928 02:22:33.426052 (+ 11871us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:33.426073 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:33.426185 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:33.428460 (+ 2275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:33.428464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:33.428488 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:33.428883 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:33.428899 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.431115 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.431120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:33.433295 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:33.433309 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.435342 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.435347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:33.435554 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:33.435596 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:22:33.435688 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 02:22:33.435815 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11738,"thread_start_us":13023,"threads_started":1}
I20260928 02:22:33.436335 24442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:33.416131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41894 (local address 127.4.140.193:36643)
0928 02:22:33.427004 (+ 10873us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:33.427014 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:33.427853 (+ 839us) server_negotiation.cc:407] Connection header received
0928 02:22:33.428094 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:33.428101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:33.428271 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:33.428450 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:33.429296 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.430974 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.432840 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.434166 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.434299 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:33.434523 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:33.435789 (+ 1266us) server_negotiation.cc:299] Negotiation successful
0928 02:22:33.436008 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10685,"thread_start_us":10737,"threads_started":1}
I20260928 02:22:33.436959 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:33.414468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39112 (local address 127.4.140.195:45175)
0928 02:22:33.427719 (+ 13251us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:33.427729 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:33.427769 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:22:33.427874 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:33.427884 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:33.428219 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:33.429103 (+ 884us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:33.429379 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.430986 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.433477 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.434851 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.434980 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:33.435204 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:33.436510 (+ 1306us) server_negotiation.cc:299] Negotiation successful
0928 02:22:33.436674 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13071,"thread_start_us":12874,"threads_started":1}
I20260928 02:22:33.440168 24360 tablet_service.cc:1467] Processing CreateTablet for tablet 2b582e9821ad42a7a0f0d6bb05b0983f (DEFAULT_TABLE table=test-table [id=11cf2bde75a5470f855ee56d1255a0e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:22:33.440168 24092 tablet_service.cc:1467] Processing CreateTablet for tablet 2b582e9821ad42a7a0f0d6bb05b0983f (DEFAULT_TABLE table=test-table [id=11cf2bde75a5470f855ee56d1255a0e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:22:33.441526 24360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b582e9821ad42a7a0f0d6bb05b0983f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:33.441565 24092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b582e9821ad42a7a0f0d6bb05b0983f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:33.449131 24443 tablet_bootstrap.cc:492] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b: Bootstrap starting.
I20260928 02:22:33.452916 24443 tablet_bootstrap.cc:654] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:33.454295 24443 log.cc:824] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:33.455235 24444 tablet_bootstrap.cc:492] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5: Bootstrap starting.
I20260928 02:22:33.457136 24445 tablet_bootstrap.cc:492] T 2b582e9821ad42a7a0f0d6bb05b0983f P 5da83fcdba3b45eab860d5f3a210e436: Bootstrap starting.
I20260928 02:22:33.459107 24443 tablet_bootstrap.cc:492] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b: No bootstrap required, opened a new log
I20260928 02:22:33.459225 24444 tablet_bootstrap.cc:654] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:33.459401 24443 ts_tablet_manager.cc:1397] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20260928 02:22:33.460470 24444 log.cc:824] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:33.461302 24445 tablet_bootstrap.cc:654] T 2b582e9821ad42a7a0f0d6bb05b0983f P 5da83fcdba3b45eab860d5f3a210e436: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:33.462491 24445 log.cc:824] T 2b582e9821ad42a7a0f0d6bb05b0983f P 5da83fcdba3b45eab860d5f3a210e436: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:33.464107 24444 tablet_bootstrap.cc:492] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5: No bootstrap required, opened a new log
I20260928 02:22:33.464370 24444 ts_tablet_manager.cc:1397] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 02:22:33.465418 24445 tablet_bootstrap.cc:492] T 2b582e9821ad42a7a0f0d6bb05b0983f P 5da83fcdba3b45eab860d5f3a210e436: No bootstrap required, opened a new log
I20260928 02:22:33.465667 24445 ts_tablet_manager.cc:1397] T 2b582e9821ad42a7a0f0d6bb05b0983f P 5da83fcdba3b45eab860d5f3a210e436: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260928 02:22:33.469905 24443 raft_consensus.cc:348] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0346a75cbba44f9862ccc51cc03560b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 36161 } } peers { permanent_uuid: "40629a2a044847b3aaca6de3889000a5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45175 } } peers { permanent_uuid: "5da83fcdba3b45eab860d5f3a210e436" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36643 } }
I20260928 02:22:33.470341 24443 raft_consensus.cc:374] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:33.470453 24443 raft_consensus.cc:729] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0346a75cbba44f9862ccc51cc03560b, State: Initialized, Role: FOLLOWER
I20260928 02:22:33.471098 24443 consensus_queue.cc:260] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0346a75cbba44f9862ccc51cc03560b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 36161 } } peers { permanent_uuid: "40629a2a044847b3aaca6de3889000a5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45175 } } peers { permanent_uuid: "5da83fcdba3b45eab860d5f3a210e436" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36643 } }
I20260928 02:22:33.471709 24444 raft_consensus.cc:348] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0346a75cbba44f9862ccc51cc03560b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 36161 } } peers { permanent_uuid: "40629a2a044847b3aaca6de3889000a5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45175 } } peers { permanent_uuid: "5da83fcdba3b45eab860d5f3a210e436" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36643 } }
I20260928 02:22:33.472229 24444 raft_consensus.cc:374] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:33.472325 24444 raft_consensus.cc:729] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40629a2a044847b3aaca6de3889000a5, State: Initialized, Role: FOLLOWER
I20260928 02:22:33.472813 24443 ts_tablet_manager.cc:1428] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
I20260928 02:22:33.472998 24444 consensus_queue.cc:260] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0346a75cbba44f9862ccc51cc03560b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 36161 } } peers { permanent_uuid: "40629a2a044847b3aaca6de3889000a5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45175 } } peers { permanent_uuid: "5da83fcdba3b45eab860d5f3a210e436" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36643 } }
I20260928 02:22:33.473282 24291 heartbeater.cc:503] Master 127.4.140.254:44391 was elected leader, sending a full tablet report...
I20260928 02:22:33.474409 24425 heartbeater.cc:503] Master 127.4.140.254:44391 was elected leader, sending a full tablet report...
I20260928 02:22:33.475118 24444 ts_tablet_manager.cc:1428] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260928 02:22:33.475867 24445 raft_consensus.cc:348] T 2b582e9821ad42a7a0f0d6bb05b0983f P 5da83fcdba3b45eab860d5f3a210e436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0346a75cbba44f9862ccc51cc03560b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 36161 } } peers { permanent_uuid: "40629a2a044847b3aaca6de3889000a5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45175 } } peers { permanent_uuid: "5da83fcdba3b45eab860d5f3a210e436" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36643 } }
I20260928 02:22:33.476343 24445 raft_consensus.cc:374] T 2b582e9821ad42a7a0f0d6bb05b0983f P 5da83fcdba3b45eab860d5f3a210e436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:33.476450 24445 raft_consensus.cc:729] T 2b582e9821ad42a7a0f0d6bb05b0983f P 5da83fcdba3b45eab860d5f3a210e436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da83fcdba3b45eab860d5f3a210e436, State: Initialized, Role: FOLLOWER
I20260928 02:22:33.477037 24445 consensus_queue.cc:260] T 2b582e9821ad42a7a0f0d6bb05b0983f P 5da83fcdba3b45eab860d5f3a210e436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0346a75cbba44f9862ccc51cc03560b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 36161 } } peers { permanent_uuid: "40629a2a044847b3aaca6de3889000a5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45175 } } peers { permanent_uuid: "5da83fcdba3b45eab860d5f3a210e436" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36643 } }
I20260928 02:22:33.479872 24445 ts_tablet_manager.cc:1428] T 2b582e9821ad42a7a0f0d6bb05b0983f P 5da83fcdba3b45eab860d5f3a210e436: Time spent starting tablet: real 0.014s user 0.002s sys 0.013s
W20260928 02:22:33.558295 24158 tablet.cc:2369] T 2b582e9821ad42a7a0f0d6bb05b0983f P 5da83fcdba3b45eab860d5f3a210e436: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:22:33.580986 24292 tablet.cc:2369] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:22:33.593546 24426 tablet.cc:2369] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:22:33.676620 24450 raft_consensus.cc:482] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:22:33.676954 24450 raft_consensus.cc:504] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0346a75cbba44f9862ccc51cc03560b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 36161 } } peers { permanent_uuid: "40629a2a044847b3aaca6de3889000a5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45175 } } peers { permanent_uuid: "5da83fcdba3b45eab860d5f3a210e436" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36643 } }
I20260928 02:22:33.678730 24450 leader_election.cc:288] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0346a75cbba44f9862ccc51cc03560b (127.4.140.194:36161), 5da83fcdba3b45eab860d5f3a210e436 (127.4.140.193:36643)
I20260928 02:22:33.684481 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:33.679193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:34535 (local address 127.4.140.194:36161)
0928 02:22:33.679378 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:33.679385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:33.679474 (+ 89us) server_negotiation.cc:407] Connection header received
0928 02:22:33.679732 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:33.679741 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:33.679850 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:33.679990 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:33.680407 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.681162 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.682752 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.683523 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.683686 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:33.683773 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:33.684089 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 02:22:33.684215 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:22:33.684485 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:33.678996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:36161 (local address 127.4.140.195:34535)
0928 02:22:33.679345 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:33.679364 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:33.679524 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:33.680016 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:33.680021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:33.680047 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:33.680260 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:33.680270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.681310 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.681315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:33.682596 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:33.682608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.683676 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.683682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:33.683870 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:33.683941 (+ 71us) client_negotiation.cc:769] Sending connection context
0928 02:22:33.684061 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 02:22:33.684196 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260928 02:22:33.685261 24442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:33.679335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:45763 (local address 127.4.140.193:36643)
0928 02:22:33.679565 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:33.679573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:33.680203 (+ 630us) server_negotiation.cc:407] Connection header received
0928 02:22:33.680427 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:33.680431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:33.680489 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:33.680592 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:33.681116 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.681799 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.683463 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.684280 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.684488 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:33.684558 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:33.684878 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 02:22:33.685016 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 02:22:33.685776 24452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:33.679176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:36643 (local address 127.4.140.195:45763)
0928 02:22:33.680085 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:33.680105 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:33.680246 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:33.680606 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:33.680610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:33.680632 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:33.680961 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:33.680969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.681961 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.681971 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:33.683295 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:33.683306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.684494 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.684502 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:33.684688 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:33.684720 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:22:33.685366 (+ 646us) client_negotiation.cc:240] Negotiation successful
0928 02:22:33.685520 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":109,"thread_start_us":450,"threads_started":1}
I20260928 02:22:33.685837 24246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b582e9821ad42a7a0f0d6bb05b0983f" candidate_uuid: "40629a2a044847b3aaca6de3889000a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0346a75cbba44f9862ccc51cc03560b" is_pre_election: true
I20260928 02:22:33.686419 24246 raft_consensus.cc:2454] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40629a2a044847b3aaca6de3889000a5 in term 0.
I20260928 02:22:33.686792 24112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b582e9821ad42a7a0f0d6bb05b0983f" candidate_uuid: "40629a2a044847b3aaca6de3889000a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da83fcdba3b45eab860d5f3a210e436" is_pre_election: true
I20260928 02:22:33.687230 24315 leader_election.cc:302] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40629a2a044847b3aaca6de3889000a5, f0346a75cbba44f9862ccc51cc03560b; no voters:
I20260928 02:22:33.687397 24112 raft_consensus.cc:2454] T 2b582e9821ad42a7a0f0d6bb05b0983f P 5da83fcdba3b45eab860d5f3a210e436 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40629a2a044847b3aaca6de3889000a5 in term 0.
I20260928 02:22:33.687641 24450 raft_consensus.cc:2781] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:22:33.687776 24450 raft_consensus.cc:482] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:22:33.687960 24450 raft_consensus.cc:3037] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:33.690464 24450 raft_consensus.cc:504] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0346a75cbba44f9862ccc51cc03560b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 36161 } } peers { permanent_uuid: "40629a2a044847b3aaca6de3889000a5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45175 } } peers { permanent_uuid: "5da83fcdba3b45eab860d5f3a210e436" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36643 } }
I20260928 02:22:33.691084 24450 leader_election.cc:288] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [CANDIDATE]: Term 1 election: Requested vote from peers f0346a75cbba44f9862ccc51cc03560b (127.4.140.194:36161), 5da83fcdba3b45eab860d5f3a210e436 (127.4.140.193:36643)
I20260928 02:22:33.691437 24246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b582e9821ad42a7a0f0d6bb05b0983f" candidate_uuid: "40629a2a044847b3aaca6de3889000a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0346a75cbba44f9862ccc51cc03560b"
I20260928 02:22:33.691592 24112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b582e9821ad42a7a0f0d6bb05b0983f" candidate_uuid: "40629a2a044847b3aaca6de3889000a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da83fcdba3b45eab860d5f3a210e436"
I20260928 02:22:33.691685 24246 raft_consensus.cc:3037] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:33.691797 24112 raft_consensus.cc:3037] T 2b582e9821ad42a7a0f0d6bb05b0983f P 5da83fcdba3b45eab860d5f3a210e436 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:33.695050 24246 raft_consensus.cc:2454] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40629a2a044847b3aaca6de3889000a5 in term 1.
I20260928 02:22:33.695042 24112 raft_consensus.cc:2454] T 2b582e9821ad42a7a0f0d6bb05b0983f P 5da83fcdba3b45eab860d5f3a210e436 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40629a2a044847b3aaca6de3889000a5 in term 1.
I20260928 02:22:33.695447 24315 leader_election.cc:302] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40629a2a044847b3aaca6de3889000a5, f0346a75cbba44f9862ccc51cc03560b; no voters:
I20260928 02:22:33.695756 24450 raft_consensus.cc:2781] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:33.696352 24450 raft_consensus.cc:686] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [term 1 LEADER]: Becoming Leader. State: Replica: 40629a2a044847b3aaca6de3889000a5, State: Running, Role: LEADER
I20260928 02:22:33.696904 24450 consensus_queue.cc:237] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0346a75cbba44f9862ccc51cc03560b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 36161 } } peers { permanent_uuid: "40629a2a044847b3aaca6de3889000a5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45175 } } peers { permanent_uuid: "5da83fcdba3b45eab860d5f3a210e436" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36643 } }
I20260928 02:22:33.702412 23967 catalog_manager.cc:5495] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40629a2a044847b3aaca6de3889000a5 (127.4.140.195). New cstate: current_term: 1 leader_uuid: "40629a2a044847b3aaca6de3889000a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0346a75cbba44f9862ccc51cc03560b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 36161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40629a2a044847b3aaca6de3889000a5" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 45175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5da83fcdba3b45eab860d5f3a210e436" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36643 } health_report { overall_health: UNKNOWN } } }
I20260928 02:22:33.728114 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:33.723048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.4.140.195:45175)
0928 02:22:33.723223 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:33.723228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:33.723297 (+ 69us) server_negotiation.cc:407] Connection header received
0928 02:22:33.723464 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:33.723468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:33.723543 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:33.723666 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:22:33.723999 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.724714 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.726152 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.726532 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.726618 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:33.727074 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:22:33.727582 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:22:33.727686 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:33.727785 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 02:22:33.727850 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:22:33.736341 24246 raft_consensus.cc:1261] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b [term 1 FOLLOWER]: Refusing update from remote peer 40629a2a044847b3aaca6de3889000a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:22:33.736341 24112 raft_consensus.cc:1261] T 2b582e9821ad42a7a0f0d6bb05b0983f P 5da83fcdba3b45eab860d5f3a210e436 [term 1 FOLLOWER]: Refusing update from remote peer 40629a2a044847b3aaca6de3889000a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:22:33.737454 24453 consensus_queue.cc:1035] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5da83fcdba3b45eab860d5f3a210e436" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 36643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:22:33.737852 24450 consensus_queue.cc:1035] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0346a75cbba44f9862ccc51cc03560b" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 36161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:22:33.750208 24456 mvcc.cc:205] Tried to move back new op lower bound from 7334142581692973056 to 7334142581547663360. Current Snapshot: MvccSnapshot[applied={T|T < 7334142581692973056}]
I20260928 02:22:33.755113 24457 mvcc.cc:205] Tried to move back new op lower bound from 7334142581692973056 to 7334142581547663360. Current Snapshot: MvccSnapshot[applied={T|T < 7334142581692973056}]
I20260928 02:22:33.768425 23966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:22:33.768760 23966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:22:33.772991 23966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5723f8d273df4f09b465e0bea7794461: Sending DeleteTablet for 3 replicas of tablet 2b582e9821ad42a7a0f0d6bb05b0983f
I20260928 02:22:33.774315 24360 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b582e9821ad42a7a0f0d6bb05b0983f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:22:33 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:39112
I20260928 02:22:33.774307 24092 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b582e9821ad42a7a0f0d6bb05b0983f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:22:33 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:41894
I20260928 02:22:33.774362 24226 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b582e9821ad42a7a0f0d6bb05b0983f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:22:33 UTC) from {username='kudu', principal='kudu/127.4.140.254@KRBTEST.COM'} at 127.0.0.1:44816
I20260928 02:22:33.777446 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 24029
I20260928 02:22:33.784809 24468 tablet_replica.cc:330] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b: stopping tablet replica
I20260928 02:22:33.785377 24466 tablet_replica.cc:330] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5: stopping tablet replica
I20260928 02:22:33.785857 24466 raft_consensus.cc:2229] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:22:33.786556 24466 raft_consensus.cc:2258] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:22:33.787263 24468 raft_consensus.cc:2229] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:22:33.787775 24468 raft_consensus.cc:2258] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:22:33.790263 24468 ts_tablet_manager.cc:1905] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:22:33 dist-test-slave-h37r krb5kdc[23922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.193@KRBTEST.COM
I20260928 02:22:33.797892 24466 ts_tablet_manager.cc:1905] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:22:33.804458 24467 tablet_replica.cc:330] T 2b582e9821ad42a7a0f0d6bb05b0983f P 5da83fcdba3b45eab860d5f3a210e436: stopping tablet replica
I20260928 02:22:33.804948 24467 raft_consensus.cc:2229] T 2b582e9821ad42a7a0f0d6bb05b0983f P 5da83fcdba3b45eab860d5f3a210e436 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:22:33.805483 24467 raft_consensus.cc:2258] T 2b582e9821ad42a7a0f0d6bb05b0983f P 5da83fcdba3b45eab860d5f3a210e436 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:22:33.806196 24468 ts_tablet_manager.cc:1918] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:22:33.806383 24468 log.cc:1196] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/2b582e9821ad42a7a0f0d6bb05b0983f
I20260928 02:22:33.806978 24468 ts_tablet_manager.cc:1939] T 2b582e9821ad42a7a0f0d6bb05b0983f P f0346a75cbba44f9862ccc51cc03560b: Deleting consensus metadata
I20260928 02:22:33.807886 24467 ts_tablet_manager.cc:1905] T 2b582e9821ad42a7a0f0d6bb05b0983f P 5da83fcdba3b45eab860d5f3a210e436: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:22:33.809020 23953 catalog_manager.cc:4841] TS f0346a75cbba44f9862ccc51cc03560b (127.4.140.194:36161): tablet 2b582e9821ad42a7a0f0d6bb05b0983f (table test-table [id=11cf2bde75a5470f855ee56d1255a0e8]) successfully deleted
I20260928 02:22:33.809506 24442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:33.778775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41898 (local address 127.4.140.193:36643)
0928 02:22:33.779008 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:33.779017 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:33.784589 (+ 5572us) server_negotiation.cc:407] Connection header received
0928 02:22:33.784795 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:33.784804 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:33.784910 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:33.785039 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:33.787271 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.788838 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.791095 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.792055 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.792211 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:33.799702 (+ 7491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:33.799729 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:33.799755 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:33.799823 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:33.805047 (+ 5224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:33.806125 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:33.806135 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:33.806138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:33.806404 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:33.807110 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:33.807120 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:33.807127 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:33.807589 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:33.807878 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:33.808715 (+ 837us) server_negotiation.cc:299] Negotiation successful
0928 02:22:33.809006 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:22:33.810864 24152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:33.811987 24466 ts_tablet_manager.cc:1918] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:22:33.812183 24466 log.cc:1196] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/2b582e9821ad42a7a0f0d6bb05b0983f
I20260928 02:22:33.812795 24466 ts_tablet_manager.cc:1939] T 2b582e9821ad42a7a0f0d6bb05b0983f P 40629a2a044847b3aaca6de3889000a5: Deleting consensus metadata
I20260928 02:22:33.814769 23952 catalog_manager.cc:4841] TS 40629a2a044847b3aaca6de3889000a5 (127.4.140.195:45175): tablet 2b582e9821ad42a7a0f0d6bb05b0983f (table test-table [id=11cf2bde75a5470f855ee56d1255a0e8]) successfully deleted
I20260928 02:22:33.917624 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 24029
W20260928 02:22:33.935458 23954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.140.193:36643 (error 108)
I20260928 02:22:33.935838 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 24163
I20260928 02:22:33.937808 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:33.937185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:36643 (local address 127.0.0.1:41912)
0928 02:22:33.937480 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:33.937610 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:36643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
W20260928 02:22:33.938057 23954 catalog_manager.cc:4584] TS 5da83fcdba3b45eab860d5f3a210e436 (127.4.140.193:36643): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b582e9821ad42a7a0f0d6bb05b0983f: Network error: Client connection negotiation failed: client connection to 127.4.140.193:36643: connect: Connection refused (error 111)
Sep 28 02:22:33 dist-test-slave-h37r krb5kdc[23922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.194@KRBTEST.COM
I20260928 02:22:33.953728 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:33.937086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44822 (local address 127.4.140.194:36161)
0928 02:22:33.937267 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:33.937273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:33.937292 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:22:33.937360 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:33.937368 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:33.937470 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:33.937585 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:33.939295 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.940086 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.942019 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:33.942389 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:33.942483 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:33.947782 (+ 5299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:33.947796 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:33.947811 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:33.947846 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:33.951015 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:33.951730 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:33.951738 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:33.951741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:33.951862 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:33.952333 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:33.952337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:33.952339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:33.952714 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:33.952952 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:33.953284 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 02:22:33.953441 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:22:33.954412 24286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:33.991940 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:33.991297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:36643 (local address 127.0.0.1:41914)
0928 02:22:33.991604 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:33.991717 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:36643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260928 02:22:33.993201 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:33.992750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:36643 (local address 127.0.0.1:41926)
0928 02:22:33.992925 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:33.993031 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:36643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 02:22:34.044828 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 24163
I20260928 02:22:34.053720 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:34.053203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:36643 (local address 127.0.0.1:41928)
0928 02:22:34.053478 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:34.053563 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:36643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 02:22:34.054792 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:34.054438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:36643 (local address 127.0.0.1:41934)
0928 02:22:34.054592 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:34.054682 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:36643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 02:22:34.068933 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 24297
Sep 28 02:22:34 dist-test-slave-h37r krb5kdc[23922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.195@KRBTEST.COM
I20260928 02:22:34.086001 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:34.069813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.4.140.195:45175)
0928 02:22:34.070032 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:34.070037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:34.070057 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:22:34.070223 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:34.070228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:34.070318 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:34.070442 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:34.071732 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:34.072603 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:34.074231 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:34.074742 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:34.074907 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:34.079787 (+ 4880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:34.079801 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:34.079817 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:34.079860 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:34.083006 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:34.083777 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:34.083785 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:34.083787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:34.084004 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:34.084395 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:34.084400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:34.084412 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:34.084792 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:34.085115 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:34.085399 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 02:22:34.085619 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 02:22:34.086691 24420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:34.135624 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:34.135109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:36643 (local address 127.0.0.1:41946)
0928 02:22:34.135341 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:34.135443 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:36643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 02:22:34.136785 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:34.136389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:36643 (local address 127.0.0.1:41950)
0928 02:22:34.136588 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:34.136662 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:36643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260928 02:22:34.211328 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 24297
I20260928 02:22:34.231460 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 23935
I20260928 02:22:34.232611 23997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:34.344121 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:34.343382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:36643 (local address 127.0.0.1:41964)
0928 02:22:34.343634 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:34.343818 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:36643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 02:22:34.344184 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 23935
2026-09-28T02:22:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4460 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T02:22:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:22:34Z Disabled control of system clock
I20260928 02:22:34.423681 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:43341
--webserver_interface=127.4.140.254
--webserver_port=0
--builtin_ntp_servers=127.4.140.212:44221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:43341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:22:34.752962 24478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:34.753245 24478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:34.753326 24478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:34.762084 24478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:22:34.762217 24478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:34.762281 24478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:34.762357 24478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:22:34.762444 24478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:22:34.772317 24478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:44221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:43341
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:43341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:34.773633 24478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:34.775544 24478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:34.786330 24483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:34.787031 24484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:34.788307 24486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:34.788944 24478 server_base.cc:1034] running on GCE node
I20260928 02:22:34.789673 24478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:34.790938 24478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:34.792132 24478 hybrid_clock.cc:648] HybridClock initialized: now 1790562154792070 us; error 59 us; skew 500 ppm
I20260928 02:22:34.792508 24478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:22:34.794863 24478 webserver.cc:460] Webserver started at http://127.4.140.254:34849/ using document root <none> and password file <none>
I20260928 02:22:34.795563 24478 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:34.795697 24478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:34.796098 24478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:34.798903 24478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "04a12048987c41009fa80577f7a76ddc"
format_stamp: "Formatted at 2026-09-28 02:22:34 on dist-test-slave-h37r"
server_key: "1b36f74f8db8854cd336159be5162e1c"
server_key_iv: "53e11e9755daa1d436cfc0fdb7a01e3a"
server_key_version: "encryptionkey@0"
I20260928 02:22:34.799698 24478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "04a12048987c41009fa80577f7a76ddc"
format_stamp: "Formatted at 2026-09-28 02:22:34 on dist-test-slave-h37r"
server_key: "1b36f74f8db8854cd336159be5162e1c"
server_key_iv: "53e11e9755daa1d436cfc0fdb7a01e3a"
server_key_version: "encryptionkey@0"
I20260928 02:22:34.804826 24478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 02:22:34.808214 24492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:34.809554 24478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:22:34.809726 24478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "04a12048987c41009fa80577f7a76ddc"
format_stamp: "Formatted at 2026-09-28 02:22:34 on dist-test-slave-h37r"
server_key: "1b36f74f8db8854cd336159be5162e1c"
server_key_iv: "53e11e9755daa1d436cfc0fdb7a01e3a"
server_key_version: "encryptionkey@0"
I20260928 02:22:34.809933 24478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:34.841603 24478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:34.842491 24478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:34.842792 24478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:34.861399 24478 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:43341
I20260928 02:22:34.861428 24543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:43341 every 8 connection(s)
I20260928 02:22:34.862733 24478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:22:34.867066 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 24478
I20260928 02:22:34.867193 24544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:34.867250 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:22:34.867815 4659 external_mini_cluster.cc:1428] Setting key 311cdd65a792af66f91c3fb1cf3c0436
I20260928 02:22:34.880759 24544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc: Bootstrap starting.
I20260928 02:22:34.881565 24547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:34.869721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60320 (local address 127.4.140.254:43341)
0928 02:22:34.870991 (+ 1270us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:34.871002 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:34.871043 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:22:34.872627 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:34.872648 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:34.873098 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:34.873497 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:34.874114 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:34.875965 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:34.877238 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:34.877904 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:34.878126 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:34.879263 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:34.879270 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:34.879282 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:34.879780 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:34.879855 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:34.879875 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:34.880055 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:34.880278 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 02:22:34.880571 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":553,"threads_started":1}
I20260928 02:22:34.883877 24544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:34.884914 24544 log.cc:824] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:34.887558 24544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc: No bootstrap required, opened a new log
I20260928 02:22:34.895309 24544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a12048987c41009fa80577f7a76ddc" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43341 } }
I20260928 02:22:34.895699 24544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:34.895829 24544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a12048987c41009fa80577f7a76ddc, State: Initialized, Role: FOLLOWER
I20260928 02:22:34.896522 24544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a12048987c41009fa80577f7a76ddc" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43341 } }
I20260928 02:22:34.896756 24544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:22:34.896905 24544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:22:34.897089 24544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:34.899190 24544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a12048987c41009fa80577f7a76ddc" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43341 } }
I20260928 02:22:34.899686 24544 leader_election.cc:302] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04a12048987c41009fa80577f7a76ddc; no voters:
I20260928 02:22:34.900338 24544 leader_election.cc:288] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:22:34.900490 24549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:34.901238 24549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc [term 1 LEADER]: Becoming Leader. State: Replica: 04a12048987c41009fa80577f7a76ddc, State: Running, Role: LEADER
I20260928 02:22:34.901750 24549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a12048987c41009fa80577f7a76ddc" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43341 } }
I20260928 02:22:34.902567 24544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:22:34.904623 24550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04a12048987c41009fa80577f7a76ddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a12048987c41009fa80577f7a76ddc" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43341 } } }
I20260928 02:22:34.904623 24551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04a12048987c41009fa80577f7a76ddc. Latest consensus state: current_term: 1 leader_uuid: "04a12048987c41009fa80577f7a76ddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a12048987c41009fa80577f7a76ddc" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43341 } } }
I20260928 02:22:34.905114 24551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:34.905112 24550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:34.906744 24554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:22:34.915665 24554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:22:34.925736 24554 catalog_manager.cc:1347] Generated new cluster ID: 723172ebbae64c959173bc3c7c6c24b0
I20260928 02:22:34.925854 24554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:22:34.943228 24554 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:22:34.945158 24554 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:22:34.953811 24554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc: Generated new TSK 0
I20260928 02:22:34.954815 24554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:22:34.969960 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.193:0
--local_ip_for_outbound_sockets=127.4.140.193
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43341
--builtin_ntp_servers=127.4.140.212:44221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:22:35.316649 24568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:35.316959 24568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:35.317106 24568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:35.324589 24568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:35.324681 24568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:35.324786 24568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.193
I20260928 02:22:35.333799 24568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:44221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.140.193
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.193
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:35.335060 24568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:35.336776 24568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:35.349778 24573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:35.349817 24574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:35.349934 24576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:35.350638 24568 server_base.cc:1034] running on GCE node
I20260928 02:22:35.351226 24568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:35.352252 24568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:35.353494 24568 hybrid_clock.cc:648] HybridClock initialized: now 1790562155353455 us; error 30 us; skew 500 ppm
I20260928 02:22:35.353808 24568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:22:35.356109 24568 webserver.cc:460] Webserver started at http://127.4.140.193:46687/ using document root <none> and password file <none>
I20260928 02:22:35.356818 24568 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:35.356904 24568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:35.357162 24568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:35.360033 24568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-0/data/instance:
uuid: "5511742253d54916bf2b95055c0220bd"
format_stamp: "Formatted at 2026-09-28 02:22:35 on dist-test-slave-h37r"
server_key: "29ab6d43f0b3d289293b53ac8860b704"
server_key_iv: "c35e46c0c0c91bf74b2ce4cc20931f17"
server_key_version: "encryptionkey@0"
I20260928 02:22:35.360746 24568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance:
uuid: "5511742253d54916bf2b95055c0220bd"
format_stamp: "Formatted at 2026-09-28 02:22:35 on dist-test-slave-h37r"
server_key: "29ab6d43f0b3d289293b53ac8860b704"
server_key_iv: "c35e46c0c0c91bf74b2ce4cc20931f17"
server_key_version: "encryptionkey@0"
I20260928 02:22:35.365689 24568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 02:22:35.369119 24582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:35.370558 24568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:22:35.370757 24568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-0/wal
uuid: "5511742253d54916bf2b95055c0220bd"
format_stamp: "Formatted at 2026-09-28 02:22:35 on dist-test-slave-h37r"
server_key: "29ab6d43f0b3d289293b53ac8860b704"
server_key_iv: "c35e46c0c0c91bf74b2ce4cc20931f17"
server_key_version: "encryptionkey@0"
I20260928 02:22:35.371002 24568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:35.386703 24568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:35.387591 24568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:35.387974 24568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:35.389004 24568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:22:35.391574 24568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:22:35.391703 24568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:35.391834 24568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:22:35.391952 24568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:35.428088 24568 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.193:37197
I20260928 02:22:35.428128 24694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.193:37197 every 8 connection(s)
I20260928 02:22:35.429504 24568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-0/data/info.pb
I20260928 02:22:35.439344 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 24568
I20260928 02:22:35.439529 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-0/wal/instance
I20260928 02:22:35.440019 4659 external_mini_cluster.cc:1428] Setting key 03814769da99f8a303117986a24a9d2e
I20260928 02:22:35.442351 24697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:35.432168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:45847 (local address 127.4.140.254:43341)
0928 02:22:35.432606 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:35.432611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:35.433853 (+ 1242us) server_negotiation.cc:407] Connection header received
0928 02:22:35.435666 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:35.435672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:35.435763 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:35.435891 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:35.436990 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:35.437713 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:35.439305 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:35.439835 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:35.440015 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:35.441480 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:35.441485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:35.441489 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:35.441506 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:35.441547 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:35.441560 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:35.441639 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:35.441911 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 02:22:35.442015 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":216,"threads_started":1}
I20260928 02:22:35.443001 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.194:0
--local_ip_for_outbound_sockets=127.4.140.194
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43341
--builtin_ntp_servers=127.4.140.212:44221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:22:35.443406 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:35.432759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43341 (local address 127.4.140.193:45847)
0928 02:22:35.433683 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:35.433724 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:35.435396 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:35.436156 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:35.436164 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:35.436246 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:22:35.436818 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:35.436835 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:35.437884 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:35.437888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:35.439154 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:35.439167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:35.440102 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:35.440111 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:35.440350 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:35.441124 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:22:35.441127 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:22:35.441178 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:22:35.441268 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:22:35.441671 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:22:35.441683 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:22:35.441690 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:22:35.441881 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 02:22:35.442268 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":263,"threads_started":1}
I20260928 02:22:35.444993 24695 heartbeater.cc:347] Connected to a master server at 127.4.140.254:43341
I20260928 02:22:35.445292 24695 heartbeater.cc:464] Registering TS with master...
I20260928 02:22:35.446080 24695 heartbeater.cc:511] Master 127.4.140.254:43341 requested a full tablet report, sending...
I20260928 02:22:35.448179 24509 ts_manager.cc:194] Registered new tserver with Master: 5511742253d54916bf2b95055c0220bd (127.4.140.193:37197)
I20260928 02:22:35.449656 24509 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.140.193:45847
W20260928 02:22:35.769341 24699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:35.769622 24699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:35.769752 24699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:35.777168 24699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:35.777271 24699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:35.777441 24699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.194
I20260928 02:22:35.786832 24699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:44221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.140.194
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.194
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:35.788208 24699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:35.790037 24699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:35.803362 24705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:35.803387 24707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:35.803350 24704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:35.803962 24699 server_base.cc:1034] running on GCE node
I20260928 02:22:35.804693 24699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:35.805644 24699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:35.806841 24699 hybrid_clock.cc:648] HybridClock initialized: now 1790562155806785 us; error 86 us; skew 500 ppm
I20260928 02:22:35.807126 24699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:22:35.809671 24699 webserver.cc:460] Webserver started at http://127.4.140.194:41679/ using document root <none> and password file <none>
I20260928 02:22:35.810336 24699 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:35.810423 24699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:35.810683 24699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:35.813858 24699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-1/data/instance:
uuid: "89741e73e28b438aa018b9278f5c2715"
format_stamp: "Formatted at 2026-09-28 02:22:35 on dist-test-slave-h37r"
server_key: "93258dd8677a25d04bda1b84463342cb"
server_key_iv: "4ad445ad85809a116fce67a00b9ccfe9"
server_key_version: "encryptionkey@0"
I20260928 02:22:35.814740 24699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance:
uuid: "89741e73e28b438aa018b9278f5c2715"
format_stamp: "Formatted at 2026-09-28 02:22:35 on dist-test-slave-h37r"
server_key: "93258dd8677a25d04bda1b84463342cb"
server_key_iv: "4ad445ad85809a116fce67a00b9ccfe9"
server_key_version: "encryptionkey@0"
I20260928 02:22:35.820129 24699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 02:22:35.823674 24713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:35.825218 24699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:22:35.825351 24699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-1/wal
uuid: "89741e73e28b438aa018b9278f5c2715"
format_stamp: "Formatted at 2026-09-28 02:22:35 on dist-test-slave-h37r"
server_key: "93258dd8677a25d04bda1b84463342cb"
server_key_iv: "4ad445ad85809a116fce67a00b9ccfe9"
server_key_version: "encryptionkey@0"
I20260928 02:22:35.825536 24699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:35.847158 24699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:35.847957 24699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:35.848230 24699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:35.849143 24699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:22:35.850540 24699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:22:35.850625 24699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:35.850695 24699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:22:35.850749 24699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:35.885581 24699 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.194:37025
I20260928 02:22:35.885671 24825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.194:37025 every 8 connection(s)
I20260928 02:22:35.886900 24699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-1/data/info.pb
I20260928 02:22:35.888037 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 24699
I20260928 02:22:35.888203 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-1/wal/instance
I20260928 02:22:35.888769 4659 external_mini_cluster.cc:1428] Setting key b90fa7f24d500ffa61f031ae6c1968e1
I20260928 02:22:35.892132 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.140.195:0
--local_ip_for_outbound_sockets=127.4.140.195
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43341
--builtin_ntp_servers=127.4.140.212:44221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:22:35.900522 24697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:35.889842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.194:40095 (local address 127.4.140.254:43341)
0928 02:22:35.890048 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:35.890053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:35.891659 (+ 1606us) server_negotiation.cc:407] Connection header received
0928 02:22:35.893502 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:35.893509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:35.893617 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:35.893727 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:35.894800 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:35.895519 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:35.897296 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:35.897685 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:35.897830 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:35.899435 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:35.899441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:35.899445 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:35.899466 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:35.899502 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:35.899516 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:35.899603 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:35.900141 (+ 538us) server_negotiation.cc:299] Negotiation successful
0928 02:22:35.900243 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 02:22:35.901656 24828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:35.890428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43341 (local address 127.4.140.194:40095)
0928 02:22:35.891510 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:35.891551 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:35.893235 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:35.893960 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:35.893969 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:35.894055 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:22:35.894621 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:35.894637 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:35.895690 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:35.895697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:35.897141 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:35.897153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:35.897922 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:35.897929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:35.898159 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:35.898952 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:22:35.898959 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:22:35.899017 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:22:35.899126 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:22:35.899628 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:22:35.899641 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:22:35.899650 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:22:35.899858 (+ 208us) client_negotiation.cc:240] Negotiation successful
0928 02:22:35.900319 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":358,"threads_started":1}
I20260928 02:22:35.903147 24826 heartbeater.cc:347] Connected to a master server at 127.4.140.254:43341
I20260928 02:22:35.903445 24826 heartbeater.cc:464] Registering TS with master...
I20260928 02:22:35.904358 24826 heartbeater.cc:511] Master 127.4.140.254:43341 requested a full tablet report, sending...
I20260928 02:22:35.906061 24509 ts_manager.cc:194] Registered new tserver with Master: 89741e73e28b438aa018b9278f5c2715 (127.4.140.194:37025)
I20260928 02:22:35.907337 24509 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.140.194:40095
W20260928 02:22:36.227200 24829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:36.227499 24829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:36.227583 24829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:36.234992 24829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:36.235109 24829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:36.235212 24829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.140.195
I20260928 02:22:36.244225 24829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:44221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.140.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.140.195
--webserver_port=0
--tserver_master_addrs=127.4.140.254:43341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.140.195
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:36.245452 24829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:36.247267 24829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:36.260074 24834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:36.260592 24837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:36.260660 24835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:36.261031 24829 server_base.cc:1034] running on GCE node
I20260928 02:22:36.261639 24829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:36.262492 24829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:36.263703 24829 hybrid_clock.cc:648] HybridClock initialized: now 1790562156263650 us; error 39 us; skew 500 ppm
I20260928 02:22:36.264102 24829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:22:36.266469 24829 webserver.cc:460] Webserver started at http://127.4.140.195:36145/ using document root <none> and password file <none>
I20260928 02:22:36.267156 24829 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:36.267268 24829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:36.267597 24829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:36.270406 24829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-2/data/instance:
uuid: "4bf7dc9073784fc5a3b05db56a517f51"
format_stamp: "Formatted at 2026-09-28 02:22:36 on dist-test-slave-h37r"
server_key: "6f498bff4aec05bfcfd392e51711049b"
server_key_iv: "5ab0e0741860b8f3b4ae194fa351aad3"
server_key_version: "encryptionkey@0"
I20260928 02:22:36.271184 24829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance:
uuid: "4bf7dc9073784fc5a3b05db56a517f51"
format_stamp: "Formatted at 2026-09-28 02:22:36 on dist-test-slave-h37r"
server_key: "6f498bff4aec05bfcfd392e51711049b"
server_key_iv: "5ab0e0741860b8f3b4ae194fa351aad3"
server_key_version: "encryptionkey@0"
I20260928 02:22:36.276437 24829 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:22:36.280050 24843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:36.281513 24829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:22:36.281708 24829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-2/wal
uuid: "4bf7dc9073784fc5a3b05db56a517f51"
format_stamp: "Formatted at 2026-09-28 02:22:36 on dist-test-slave-h37r"
server_key: "6f498bff4aec05bfcfd392e51711049b"
server_key_iv: "5ab0e0741860b8f3b4ae194fa351aad3"
server_key_version: "encryptionkey@0"
I20260928 02:22:36.281934 24829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:36.309957 24829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:36.310962 24829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:36.311275 24829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:36.312326 24829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:22:36.314037 24829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:22:36.314157 24829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:36.314292 24829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:22:36.314384 24829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:36.348544 24829 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.195:43359
I20260928 02:22:36.348634 24955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.195:43359 every 8 connection(s)
I20260928 02:22:36.349985 24829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-2/data/info.pb
I20260928 02:22:36.359694 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 24829
I20260928 02:22:36.359956 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-2/wal/instance
I20260928 02:22:36.360445 4659 external_mini_cluster.cc:1428] Setting key 4563a1d560c62f95e5f9b8cf3d3b2eb1
I20260928 02:22:36.362912 24697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.352751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.195:40349 (local address 127.4.140.254:43341)
0928 02:22:36.352931 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:36.352936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:36.354347 (+ 1411us) server_negotiation.cc:407] Connection header received
0928 02:22:36.356315 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:36.356321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:36.356417 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:36.356541 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:36.357660 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.358402 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.360059 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.360464 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.360614 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:36.362011 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:36.362016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:36.362019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:36.362038 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:36.362071 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:36.362083 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:36.362181 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:36.362416 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 02:22:36.362527 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:22:36.364297 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.353298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:43341 (local address 127.4.140.195:40349)
0928 02:22:36.354182 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:36.354217 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:36.356034 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:36.356778 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:36.356786 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:36.356877 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:22:36.357480 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:36.357497 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.358573 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.358581 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:36.359826 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:36.359868 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.360701 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.360708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:36.360942 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:36.361686 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:22:36.361688 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:22:36.361737 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:22:36.361814 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:22:36.362180 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:22:36.362192 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:22:36.362202 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:22:36.362392 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 02:22:36.362797 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":265,"threads_started":1}
I20260928 02:22:36.366012 24956 heartbeater.cc:347] Connected to a master server at 127.4.140.254:43341
I20260928 02:22:36.366333 24956 heartbeater.cc:464] Registering TS with master...
I20260928 02:22:36.367108 24956 heartbeater.cc:511] Master 127.4.140.254:43341 requested a full tablet report, sending...
I20260928 02:22:36.368588 24509 ts_manager.cc:194] Registered new tserver with Master: 4bf7dc9073784fc5a3b05db56a517f51 (127.4.140.195:43359)
I20260928 02:22:36.369377 24509 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.140.195:40349
I20260928 02:22:36.376464 4659 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:22:36.387223 24697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.380713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42978 (local address 127.4.140.254:43341)
0928 02:22:36.380886 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:36.380891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:36.381311 (+ 420us) server_negotiation.cc:407] Connection header received
0928 02:22:36.381457 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:36.381461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:36.381543 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:36.381628 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:36.382125 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.382822 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.384276 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.384680 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.384771 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:36.386307 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:36.386313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:36.386318 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:36.386341 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:36.386403 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:36.386418 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:36.386530 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:36.386762 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 02:22:36.386879 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:22:36.391943 24509 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:22:36.401101 24509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:22:36.434602 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.413947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:43359 (local address 127.0.0.1:55440)
0928 02:22:36.423444 (+ 9497us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:36.423517 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:36.423792 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:36.424788 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:36.424800 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:36.424860 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:36.425356 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:36.425374 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.428272 (+ 2898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.428281 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:36.431112 (+ 2831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:36.431129 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.433503 (+ 2374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.433509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:36.433751 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:36.433825 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 02:22:36.433936 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 02:22:36.434249 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9143,"mutex_wait_us":83,"thread_start_us":9001,"threads_started":1}
I20260928 02:22:36.435331 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.414121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55440 (local address 127.4.140.195:43359)
0928 02:22:36.422904 (+ 8783us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:36.422929 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:36.424066 (+ 1137us) server_negotiation.cc:407] Connection header received
0928 02:22:36.424178 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:36.424189 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:36.424472 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:36.425629 (+ 1157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:36.425992 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.428060 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.431418 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.433293 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.434128 (+ 835us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:36.434496 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:36.434589 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 02:22:36.434847 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8532,"thread_start_us":8337,"threads_started":1}
I20260928 02:22:36.439352 24970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.414123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:37197 (local address 127.0.0.1:50062)
0928 02:22:36.424780 (+ 10657us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:36.424809 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:36.424962 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:36.431234 (+ 6272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:36.431238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:36.431267 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:36.431596 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:36.431609 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.433640 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.433645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:36.436932 (+ 3287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:36.436951 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.438594 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.438601 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:36.438767 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:36.438811 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 02:22:36.438909 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 02:22:36.439043 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10475,"thread_start_us":16146,"threads_started":1}
I20260928 02:22:36.439704 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.414684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50062 (local address 127.4.140.193:37197)
0928 02:22:36.430121 (+ 15437us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:36.430130 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:36.430161 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:22:36.430452 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:36.430461 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:36.430737 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:36.430945 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:36.431779 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.433469 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.437189 (+ 3720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.438353 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.438552 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:36.438821 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:36.438952 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 02:22:36.439137 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15230,"thread_start_us":15582,"threads_started":1}
I20260928 02:22:36.442759 24630 tablet_service.cc:1467] Processing CreateTablet for tablet d366de76eda147518d45a82d5acba6f3 (DEFAULT_TABLE table=test-table [id=5a82b0bc1f1546518c3dc43869e0feb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:22:36.442369 24891 tablet_service.cc:1467] Processing CreateTablet for tablet d366de76eda147518d45a82d5acba6f3 (DEFAULT_TABLE table=test-table [id=5a82b0bc1f1546518c3dc43869e0feb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:22:36.444051 24630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d366de76eda147518d45a82d5acba6f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:36.444306 24891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d366de76eda147518d45a82d5acba6f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:36.452598 24695 heartbeater.cc:503] Master 127.4.140.254:43341 was elected leader, sending a full tablet report...
I20260928 02:22:36.456384 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.424135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:37025 (local address 127.0.0.1:40866)
0928 02:22:36.434951 (+ 10816us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:36.434971 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:36.435111 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:36.449118 (+ 14007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:36.449124 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:36.449155 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:36.449345 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:36.449357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.451457 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.451463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:36.453341 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:36.453356 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.455562 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.455568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:36.455766 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:36.455805 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 02:22:36.455874 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:22:36.456056 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10714,"thread_start_us":12831,"threads_started":1}
I20260928 02:22:36.457469 24973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.424347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40866 (local address 127.4.140.194:37025)
0928 02:22:36.448405 (+ 24058us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:36.448415 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:36.448448 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:22:36.448587 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:36.448596 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:36.448862 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:36.449529 (+ 667us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:36.449746 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.451281 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.453568 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.455369 (+ 1801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.456635 (+ 1266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:36.456887 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:36.456960 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 02:22:36.457144 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23881,"thread_start_us":23769,"threads_started":1}
I20260928 02:22:36.458356 24974 tablet_bootstrap.cc:492] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd: Bootstrap starting.
I20260928 02:22:36.459300 24975 tablet_bootstrap.cc:492] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51: Bootstrap starting.
I20260928 02:22:36.461278 24761 tablet_service.cc:1467] Processing CreateTablet for tablet d366de76eda147518d45a82d5acba6f3 (DEFAULT_TABLE table=test-table [id=5a82b0bc1f1546518c3dc43869e0feb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:22:36.462610 24761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d366de76eda147518d45a82d5acba6f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:36.462960 24975 tablet_bootstrap.cc:654] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:36.463093 24974 tablet_bootstrap.cc:654] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:36.464305 24975 log.cc:824] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:36.464305 24974 log.cc:824] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:36.467260 24975 tablet_bootstrap.cc:492] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51: No bootstrap required, opened a new log
I20260928 02:22:36.467540 24975 ts_tablet_manager.cc:1397] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 02:22:36.467540 24974 tablet_bootstrap.cc:492] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd: No bootstrap required, opened a new log
I20260928 02:22:36.467821 24974 ts_tablet_manager.cc:1397] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 02:22:36.475375 24975 raft_consensus.cc:348] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf7dc9073784fc5a3b05db56a517f51" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43359 } } peers { permanent_uuid: "5511742253d54916bf2b95055c0220bd" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 37197 } } peers { permanent_uuid: "89741e73e28b438aa018b9278f5c2715" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37025 } }
I20260928 02:22:36.475802 24975 raft_consensus.cc:374] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:36.475894 24975 raft_consensus.cc:729] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bf7dc9073784fc5a3b05db56a517f51, State: Initialized, Role: FOLLOWER
I20260928 02:22:36.476539 24975 consensus_queue.cc:260] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf7dc9073784fc5a3b05db56a517f51" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43359 } } peers { permanent_uuid: "5511742253d54916bf2b95055c0220bd" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 37197 } } peers { permanent_uuid: "89741e73e28b438aa018b9278f5c2715" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37025 } }
I20260928 02:22:36.477528 24978 tablet_bootstrap.cc:492] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715: Bootstrap starting.
I20260928 02:22:36.478282 24974 raft_consensus.cc:348] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf7dc9073784fc5a3b05db56a517f51" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43359 } } peers { permanent_uuid: "5511742253d54916bf2b95055c0220bd" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 37197 } } peers { permanent_uuid: "89741e73e28b438aa018b9278f5c2715" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37025 } }
I20260928 02:22:36.478672 24956 heartbeater.cc:503] Master 127.4.140.254:43341 was elected leader, sending a full tablet report...
I20260928 02:22:36.478729 24974 raft_consensus.cc:374] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:36.478916 24974 raft_consensus.cc:729] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5511742253d54916bf2b95055c0220bd, State: Initialized, Role: FOLLOWER
I20260928 02:22:36.479504 24974 consensus_queue.cc:260] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf7dc9073784fc5a3b05db56a517f51" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43359 } } peers { permanent_uuid: "5511742253d54916bf2b95055c0220bd" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 37197 } } peers { permanent_uuid: "89741e73e28b438aa018b9278f5c2715" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37025 } }
I20260928 02:22:36.481504 24978 tablet_bootstrap.cc:654] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:36.482870 24978 log.cc:824] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:36.484302 24975 ts_tablet_manager.cc:1428] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51: Time spent starting tablet: real 0.017s user 0.006s sys 0.007s
I20260928 02:22:36.485483 24974 ts_tablet_manager.cc:1428] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd: Time spent starting tablet: real 0.018s user 0.011s sys 0.005s
I20260928 02:22:36.488986 24978 tablet_bootstrap.cc:492] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715: No bootstrap required, opened a new log
I20260928 02:22:36.489267 24978 ts_tablet_manager.cc:1397] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260928 02:22:36.496155 24978 raft_consensus.cc:348] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf7dc9073784fc5a3b05db56a517f51" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43359 } } peers { permanent_uuid: "5511742253d54916bf2b95055c0220bd" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 37197 } } peers { permanent_uuid: "89741e73e28b438aa018b9278f5c2715" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37025 } }
I20260928 02:22:36.496610 24978 raft_consensus.cc:374] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:36.496731 24978 raft_consensus.cc:729] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89741e73e28b438aa018b9278f5c2715, State: Initialized, Role: FOLLOWER
I20260928 02:22:36.497251 24978 consensus_queue.cc:260] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf7dc9073784fc5a3b05db56a517f51" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43359 } } peers { permanent_uuid: "5511742253d54916bf2b95055c0220bd" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 37197 } } peers { permanent_uuid: "89741e73e28b438aa018b9278f5c2715" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37025 } }
I20260928 02:22:36.497828 24980 raft_consensus.cc:482] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:22:36.498090 24980 raft_consensus.cc:504] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf7dc9073784fc5a3b05db56a517f51" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43359 } } peers { permanent_uuid: "5511742253d54916bf2b95055c0220bd" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 37197 } } peers { permanent_uuid: "89741e73e28b438aa018b9278f5c2715" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37025 } }
I20260928 02:22:36.498342 24826 heartbeater.cc:503] Master 127.4.140.254:43341 was elected leader, sending a full tablet report...
I20260928 02:22:36.498996 24978 ts_tablet_manager.cc:1428] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260928 02:22:36.499759 24980 leader_election.cc:288] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bf7dc9073784fc5a3b05db56a517f51 (127.4.140.195:43359), 89741e73e28b438aa018b9278f5c2715 (127.4.140.194:37025)
I20260928 02:22:36.509296 24973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.501329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:48983 (local address 127.4.140.194:37025)
0928 02:22:36.501500 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:36.501505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:36.501710 (+ 205us) server_negotiation.cc:407] Connection header received
0928 02:22:36.502088 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:36.502093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:36.502172 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:36.502270 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:36.502947 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.503996 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.507120 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.508312 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.508469 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:36.508542 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:36.508933 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 02:22:36.509048 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:22:36.509296 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.500193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.140.193:47585 (local address 127.4.140.195:43359)
0928 02:22:36.500397 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:36.500404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:36.501547 (+ 1143us) server_negotiation.cc:407] Connection header received
0928 02:22:36.501905 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:36.501910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:36.501982 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:36.502082 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:22:36.502745 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.503959 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.507123 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.508421 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.508560 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:36.508621 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:36.508924 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 02:22:36.509048 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:22:36.509516 24983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.500123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.195:43359 (local address 127.4.140.193:47585)
0928 02:22:36.500900 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:36.500930 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:36.501646 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:36.502097 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:36.502102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:36.502124 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:36.502577 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:36.502615 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.504106 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.504112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:36.506904 (+ 2792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:36.506917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.508575 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.508581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:36.508777 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:36.508811 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:22:36.508883 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:22:36.509111 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":200,"thread_start_us":376,"threads_started":1}
I20260928 02:22:36.509848 24984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.501016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.194:37025 (local address 127.4.140.193:48983)
0928 02:22:36.501599 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:36.501616 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:22:36.501744 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:22:36.502309 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:22:36.502319 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:22:36.502340 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:22:36.502795 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:36.502813 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.504113 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.504116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:36.506902 (+ 2786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:22:36.506917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.508494 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.508501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:22:36.508688 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:36.508787 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 02:22:36.508883 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:22:36.509111 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":349,"thread_start_us":283,"threads_started":1}
I20260928 02:22:36.510560 24911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d366de76eda147518d45a82d5acba6f3" candidate_uuid: "5511742253d54916bf2b95055c0220bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bf7dc9073784fc5a3b05db56a517f51" is_pre_election: true
I20260928 02:22:36.510867 24781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d366de76eda147518d45a82d5acba6f3" candidate_uuid: "5511742253d54916bf2b95055c0220bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89741e73e28b438aa018b9278f5c2715" is_pre_election: true
I20260928 02:22:36.511220 24911 raft_consensus.cc:2454] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5511742253d54916bf2b95055c0220bd in term 0.
I20260928 02:22:36.511400 24781 raft_consensus.cc:2454] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5511742253d54916bf2b95055c0220bd in term 0.
I20260928 02:22:36.512009 24585 leader_election.cc:302] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bf7dc9073784fc5a3b05db56a517f51, 5511742253d54916bf2b95055c0220bd; no voters:
I20260928 02:22:36.512444 24980 raft_consensus.cc:2781] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:22:36.512594 24980 raft_consensus.cc:482] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:22:36.512704 24980 raft_consensus.cc:3037] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:36.515116 24980 raft_consensus.cc:504] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf7dc9073784fc5a3b05db56a517f51" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43359 } } peers { permanent_uuid: "5511742253d54916bf2b95055c0220bd" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 37197 } } peers { permanent_uuid: "89741e73e28b438aa018b9278f5c2715" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37025 } }
I20260928 02:22:36.515684 24980 leader_election.cc:288] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [CANDIDATE]: Term 1 election: Requested vote from peers 4bf7dc9073784fc5a3b05db56a517f51 (127.4.140.195:43359), 89741e73e28b438aa018b9278f5c2715 (127.4.140.194:37025)
I20260928 02:22:36.516132 24911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d366de76eda147518d45a82d5acba6f3" candidate_uuid: "5511742253d54916bf2b95055c0220bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bf7dc9073784fc5a3b05db56a517f51"
I20260928 02:22:36.516175 24781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d366de76eda147518d45a82d5acba6f3" candidate_uuid: "5511742253d54916bf2b95055c0220bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89741e73e28b438aa018b9278f5c2715"
I20260928 02:22:36.516338 24911 raft_consensus.cc:3037] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:36.516350 24781 raft_consensus.cc:3037] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:36.518484 24911 raft_consensus.cc:2454] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5511742253d54916bf2b95055c0220bd in term 1.
I20260928 02:22:36.518534 24781 raft_consensus.cc:2454] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5511742253d54916bf2b95055c0220bd in term 1.
I20260928 02:22:36.518867 24585 leader_election.cc:302] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bf7dc9073784fc5a3b05db56a517f51, 5511742253d54916bf2b95055c0220bd; no voters:
I20260928 02:22:36.519163 24980 raft_consensus.cc:2781] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:36.519613 24980 raft_consensus.cc:686] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [term 1 LEADER]: Becoming Leader. State: Replica: 5511742253d54916bf2b95055c0220bd, State: Running, Role: LEADER
I20260928 02:22:36.520153 24980 consensus_queue.cc:237] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf7dc9073784fc5a3b05db56a517f51" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43359 } } peers { permanent_uuid: "5511742253d54916bf2b95055c0220bd" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 37197 } } peers { permanent_uuid: "89741e73e28b438aa018b9278f5c2715" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37025 } }
I20260928 02:22:36.524708 24509 catalog_manager.cc:5495] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 5511742253d54916bf2b95055c0220bd (127.4.140.193). New cstate: current_term: 1 leader_uuid: "5511742253d54916bf2b95055c0220bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf7dc9073784fc5a3b05db56a517f51" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5511742253d54916bf2b95055c0220bd" member_type: VOTER last_known_addr { host: "127.4.140.193" port: 37197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89741e73e28b438aa018b9278f5c2715" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20260928 02:22:36.542559 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.537460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50072 (local address 127.4.140.193:37197)
0928 02:22:36.537705 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:36.537709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:36.537727 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:22:36.537890 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:36.537894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:36.537981 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:36.538117 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:22:36.538414 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.539119 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.540552 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.540958 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.541059 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:36.541474 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:22:36.541977 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:22:36.542107 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:36.542185 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 02:22:36.542250 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:22:36.550791 24911 raft_consensus.cc:1261] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51 [term 1 FOLLOWER]: Refusing update from remote peer 5511742253d54916bf2b95055c0220bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:22:36.550853 24781 raft_consensus.cc:1261] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715 [term 1 FOLLOWER]: Refusing update from remote peer 5511742253d54916bf2b95055c0220bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:22:36.551748 24985 consensus_queue.cc:1035] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bf7dc9073784fc5a3b05db56a517f51" member_type: VOTER last_known_addr { host: "127.4.140.195" port: 43359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:22:36.552188 24980 consensus_queue.cc:1035] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "89741e73e28b438aa018b9278f5c2715" member_type: VOTER last_known_addr { host: "127.4.140.194" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:22:36.578024 24509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:22:36.578315 24509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:22:36.581833 24509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04a12048987c41009fa80577f7a76ddc: Sending DeleteTablet for 3 replicas of tablet d366de76eda147518d45a82d5acba6f3
I20260928 02:22:36.582922 24630 tablet_service.cc:1514] Processing DeleteTablet for tablet d366de76eda147518d45a82d5acba6f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:22:36 UTC) from {username='slave'} at 127.0.0.1:50062
I20260928 02:22:36.583061 24891 tablet_service.cc:1514] Processing DeleteTablet for tablet d366de76eda147518d45a82d5acba6f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:22:36 UTC) from {username='slave'} at 127.0.0.1:55440
I20260928 02:22:36.583710 24761 tablet_service.cc:1514] Processing DeleteTablet for tablet d366de76eda147518d45a82d5acba6f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:22:36 UTC) from {username='slave'} at 127.0.0.1:40866
I20260928 02:22:36.584031 24998 tablet_replica.cc:330] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd: stopping tablet replica
I20260928 02:22:36.585403 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 24568
I20260928 02:22:36.593992 24998 raft_consensus.cc:2229] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:22:36.595482 24998 raft_consensus.cc:2258] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:22:36.599680 24998 ts_tablet_manager.cc:1905] T d366de76eda147518d45a82d5acba6f3 P 5511742253d54916bf2b95055c0220bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:22:36.600672 24999 tablet_replica.cc:330] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51: stopping tablet replica
I20260928 02:22:36.601145 24999 raft_consensus.cc:2229] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:22:36.602854 25000 tablet_replica.cc:330] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715: stopping tablet replica
I20260928 02:22:36.602931 24999 raft_consensus.cc:2258] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:22:36.603308 25000 raft_consensus.cc:2229] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:22:36.603848 25000 raft_consensus.cc:2258] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:22:36.605201 24999 ts_tablet_manager.cc:1905] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:22:36.606051 25000 ts_tablet_manager.cc:1905] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:22:36.609735 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.587261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50076 (local address 127.4.140.193:37197)
0928 02:22:36.587495 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:36.587504 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:36.601164 (+ 13660us) server_negotiation.cc:407] Connection header received
0928 02:22:36.601585 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:36.601595 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:36.601738 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:36.601841 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:36.602821 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.604088 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.605819 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.606438 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.606542 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:36.608039 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:36.608054 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:36.608071 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:36.608831 (+ 760us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:36.608885 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:36.608907 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:36.609002 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:36.609269 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 02:22:36.609374 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:22:36.610888 24690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:36.613515 25000 ts_tablet_manager.cc:1918] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:22:36.613768 25000 log.cc:1196] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-1/wal/wals/d366de76eda147518d45a82d5acba6f3
I20260928 02:22:36.614008 24999 ts_tablet_manager.cc:1918] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:22:36.614207 24999 log.cc:1196] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51: Deleting WAL directory at /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561835360118-4659-0/minicluster-data/ts-2/wal/wals/d366de76eda147518d45a82d5acba6f3
I20260928 02:22:36.614388 25000 ts_tablet_manager.cc:1939] T d366de76eda147518d45a82d5acba6f3 P 89741e73e28b438aa018b9278f5c2715: Deleting consensus metadata
I20260928 02:22:36.614909 24999 ts_tablet_manager.cc:1939] T d366de76eda147518d45a82d5acba6f3 P 4bf7dc9073784fc5a3b05db56a517f51: Deleting consensus metadata
I20260928 02:22:36.616185 24495 catalog_manager.cc:4841] TS 89741e73e28b438aa018b9278f5c2715 (127.4.140.194:37025): tablet d366de76eda147518d45a82d5acba6f3 (table test-table [id=5a82b0bc1f1546518c3dc43869e0feb7]) successfully deleted
I20260928 02:22:36.616952 24495 catalog_manager.cc:4841] TS 4bf7dc9073784fc5a3b05db56a517f51 (127.4.140.195:43359): tablet d366de76eda147518d45a82d5acba6f3 (table test-table [id=5a82b0bc1f1546518c3dc43869e0feb7]) successfully deleted
I20260928 02:22:36.711858 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 24568
I20260928 02:22:36.731140 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 24699
W20260928 02:22:36.731241 24494 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.140.193:37197 (error 108)
I20260928 02:22:36.733378 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.732864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:37197 (local address 127.0.0.1:50084)
0928 02:22:36.733060 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:36.733154 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:37197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260928 02:22:36.733626 24494 catalog_manager.cc:4584] TS 5511742253d54916bf2b95055c0220bd (127.4.140.193:37197): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d366de76eda147518d45a82d5acba6f3: Network error: Client connection negotiation failed: client connection to 127.4.140.193:37197: connect: Connection refused (error 111)
I20260928 02:22:36.738570 24973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.732110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.4.140.194:37025)
0928 02:22:36.732320 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:36.732326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:36.732374 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:22:36.732557 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:36.732565 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:36.732691 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:36.732798 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:36.733265 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.734208 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.735464 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.736175 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.736316 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:36.737248 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:36.737256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:36.737270 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:36.737911 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:36.737959 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:36.737980 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:36.738071 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:36.738217 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 02:22:36.738305 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:22:36.739269 24821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:36.787320 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.786782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:37197 (local address 127.0.0.1:50092)
0928 02:22:36.787020 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:36.787131 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:37197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 02:22:36.788600 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.788123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:37197 (local address 127.0.0.1:50108)
0928 02:22:36.788313 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:36.788401 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:37197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 02:22:36.831436 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 24699
I20260928 02:22:36.848344 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 24829
I20260928 02:22:36.849247 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.848723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:37197 (local address 127.0.0.1:50120)
0928 02:22:36.848961 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:36.849067 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:37197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 02:22:36.850387 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.849970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:37197 (local address 127.0.0.1:50134)
0928 02:22:36.850138 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:36.850208 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:37197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 02:22:36.855597 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.849209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55444 (local address 127.4.140.195:43359)
0928 02:22:36.849392 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:36.849399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:36.849459 (+ 60us) server_negotiation.cc:407] Connection header received
0928 02:22:36.849717 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:36.849723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:36.849849 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:36.849935 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:36.850406 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.851227 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.852668 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:36.853091 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:36.853188 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:36.854200 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:36.854208 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:36.854222 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:36.854847 (+ 625us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:36.854893 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:36.854911 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:36.855015 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:36.855122 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 02:22:36.855229 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:22:36.856271 24951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:36.931034 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.930435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:37197 (local address 127.0.0.1:50146)
0928 02:22:36.930682 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:36.930856 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:37197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 02:22:36.932235 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:36.931724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.193:37197 (local address 127.0.0.1:50156)
0928 02:22:36.931971 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 02:22:36.932076 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.193:37197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260928 02:22:36.948094 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 24829
I20260928 02:22:36.964507 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 24478
I20260928 02:22:36.965597 24539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:37.074728 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 24478
2026-09-28T02:22:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2733 ms)
[----------] 6 tests from EncryptionPolicyTest (21811 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:22:37 dist-test-slave-h37r krb5kdc[25009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:22:37 dist-test-slave-h37r krb5kdc[25009](info): set up 2 sockets
Sep 28 02:22:37 dist-test-slave-h37r krb5kdc[25009](info): commencing operation
krb5kdc: starting...
W20260928 02:22:38.252519 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:22:38 dist-test-slave-h37r krb5kdc[25009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:22:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:22:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:22:38.519466 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36513
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:38.860257 25022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:38.860559 25022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:38.860651 25022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:38.867985 25022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:22:38.868121 25022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:38.868168 25022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:38.868222 25022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:22:38.868265 25022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:22:38.878665 25022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36513
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:38.880034 25022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:38.881349 25022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:38.894011 25027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:38.894022 25030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:38.894066 25028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:38.894953 25022 server_base.cc:1034] running on GCE node
I20260928 02:22:38.895896 25022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:38.897483 25022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:38.898701 25022 hybrid_clock.cc:648] HybridClock initialized: now 1790562158898650 us; error 40 us; skew 500 ppm
I20260928 02:22:38.899089 25022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:38 dist-test-slave-h37r krb5kdc[25009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:38.912379 25022 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:22:38.914142 25022 webserver.cc:460] Webserver started at http://127.0.0.1:39937/ using document root <none> and password file <none>
I20260928 02:22:38.914902 25022 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:38.915073 25022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:38.915414 25022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:38.918721 25022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "6370199ded464e7bb03630c03006b9c3"
format_stamp: "Formatted at 2026-09-28 02:22:38 on dist-test-slave-h37r"
server_key: "29e235107ebae719ebed61fe49e470ab"
server_key_iv: "2dc3100e3b8dd1063985320ccd5d838a"
server_key_version: "encryptionkey@0"
I20260928 02:22:38.919555 25022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "6370199ded464e7bb03630c03006b9c3"
format_stamp: "Formatted at 2026-09-28 02:22:38 on dist-test-slave-h37r"
server_key: "29e235107ebae719ebed61fe49e470ab"
server_key_iv: "2dc3100e3b8dd1063985320ccd5d838a"
server_key_version: "encryptionkey@0"
I20260928 02:22:38.925109 25022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 02:22:38.928808 25037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:38.930217 25022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 02:22:38.930367 25022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "6370199ded464e7bb03630c03006b9c3"
format_stamp: "Formatted at 2026-09-28 02:22:38 on dist-test-slave-h37r"
server_key: "29e235107ebae719ebed61fe49e470ab"
server_key_iv: "2dc3100e3b8dd1063985320ccd5d838a"
server_key_version: "encryptionkey@0"
I20260928 02:22:38.930543 25022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:38.962940 25022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:38.967188 25022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:38.967427 25022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:38.986474 25022 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36513
I20260928 02:22:38.986509 25088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36513 every 8 connection(s)
I20260928 02:22:38.988056 25022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:22:38.991998 25089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:38.996862 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 25022
I20260928 02:22:38.997080 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:22:38.997716 4659 external_mini_cluster.cc:1428] Setting key 03c81f3a5490cd33c1c74bd463ce5a81
I20260928 02:22:39.004182 25089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3: Bootstrap starting.
I20260928 02:22:39.008024 25089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:39.009033 25089 log.cc:824] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:39.011830 25089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3: No bootstrap required, opened a new log
Sep 28 02:22:39 dist-test-slave-h37r krb5kdc[25009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:22:39.019655 25089 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6370199ded464e7bb03630c03006b9c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } }
I20260928 02:22:39.020149 25089 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:39.020272 25089 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6370199ded464e7bb03630c03006b9c3, State: Initialized, Role: FOLLOWER
I20260928 02:22:39.020967 25089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6370199ded464e7bb03630c03006b9c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } }
I20260928 02:22:39.021229 25089 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:22:39.021370 25089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:22:39.021588 25089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:39.022862 25092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:38.999832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53333 (local address 127.0.0.1:36513)
0928 02:22:39.000823 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:39.000838 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:39.000899 (+ 61us) server_negotiation.cc:407] Connection header received
0928 02:22:39.003278 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:39.003306 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:39.003827 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:39.004308 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:39.005681 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:39.007706 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:39.009311 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:39.010176 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:39.010375 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:39.014596 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:39.014605 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:39.014624 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:39.014666 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:39.018195 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:39.019083 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:39.019090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:39.019097 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:39.019265 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:39.019739 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:39.019751 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:39.019754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:39.020574 (+ 820us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:39.020959 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:39.021437 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 02:22:39.021798 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":359,"threads_started":1}
I20260928 02:22:39.024413 25089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6370199ded464e7bb03630c03006b9c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } }
I20260928 02:22:39.025007 25089 leader_election.cc:302] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6370199ded464e7bb03630c03006b9c3; no voters:
I20260928 02:22:39.025672 25089 leader_election.cc:288] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:22:39.025790 25094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:39.026638 25094 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6370199ded464e7bb03630c03006b9c3, State: Running, Role: LEADER
I20260928 02:22:39.027314 25094 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6370199ded464e7bb03630c03006b9c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } }
I20260928 02:22:39.027514 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:22:39.028036 25089 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:22:39.029978 25095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6370199ded464e7bb03630c03006b9c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6370199ded464e7bb03630c03006b9c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } } }
I20260928 02:22:39.030371 25095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:39.030937 25096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6370199ded464e7bb03630c03006b9c3. Latest consensus state: current_term: 1 leader_uuid: "6370199ded464e7bb03630c03006b9c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6370199ded464e7bb03630c03006b9c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } } }
I20260928 02:22:39.031242 25096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:39.037233 25099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:22:39.044811 25099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:22:39.058468 25092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:39.039495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41377 (local address 127.0.0.1:36513)
0928 02:22:39.039698 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:39.039705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:39.039981 (+ 276us) server_negotiation.cc:407] Connection header received
0928 02:22:39.040243 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:39.040255 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:39.040359 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:39.040473 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:39.042572 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:39.043833 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:39.045805 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:39.046529 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:39.046671 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:39.050070 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:39.050078 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:39.050083 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:39.050124 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:39.054065 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:39.055101 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:39.055106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:39.055109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:39.055257 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:39.055856 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:39.055861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:39.055864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:39.056358 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:39.056649 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:39.057898 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0928 02:22:39.058083 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:22:39.069363 25099 catalog_manager.cc:1347] Generated new cluster ID: 2b382b41dd304db7a6893bdc65743cc2
I20260928 02:22:39.069501 25099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:22:39.079939 25099 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:22:39.081311 25099 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:22:39.092348 25099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6370199ded464e7bb03630c03006b9c3: Generated new TSK 0
I20260928 02:22:39.093935 25099 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:22:39.108497 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 25022
I20260928 02:22:39.110265 25084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:39.204788 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 25022
W20260928 02:22:39.222644 25007 connection.cc:535] client connection to 127.0.0.1:36513 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36513): Connection reset by peer (error 104)
2026-09-28T02:22:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2120 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:22:39 dist-test-slave-h37r krb5kdc[25121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:22:39 dist-test-slave-h37r krb5kdc[25121](info): set up 2 sockets
Sep 28 02:22:39 dist-test-slave-h37r krb5kdc[25121](info): commencing operation
krb5kdc: starting...
W20260928 02:22:40.308475 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:22:40 dist-test-slave-h37r krb5kdc[25121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:22:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:22:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:22:40.566946 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42385
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:40.900068 25134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:40.900422 25134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:40.900592 25134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:40.907951 25134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:22:40.908113 25134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:40.908223 25134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:40.908331 25134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:22:40.908411 25134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:22:40.918516 25134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42385
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:40.919870 25134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:40.921402 25134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:40.933297 25139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:40.933341 25140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:40.933444 25142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:40.934365 25134 server_base.cc:1034] running on GCE node
I20260928 02:22:40.935182 25134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:40.936705 25134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:40.937966 25134 hybrid_clock.cc:648] HybridClock initialized: now 1790562160937912 us; error 63 us; skew 500 ppm
I20260928 02:22:40.938340 25134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:40 dist-test-slave-h37r krb5kdc[25121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:40.941543 25134 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:22:40.942972 25134 webserver.cc:460] Webserver started at http://127.0.0.1:37827/ using document root <none> and password file <none>
I20260928 02:22:40.943629 25134 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:40.943733 25134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:40.944034 25134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:40.946902 25134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "765a581f57854f509e8d4a818333e25d"
format_stamp: "Formatted at 2026-09-28 02:22:40 on dist-test-slave-h37r"
server_key: "c3a25b90fabc13fc5699f7cdadc432b5"
server_key_iv: "8bf0d61471662d4f6a0e5f0a9a4c6b19"
server_key_version: "encryptionkey@0"
I20260928 02:22:40.947605 25134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "765a581f57854f509e8d4a818333e25d"
format_stamp: "Formatted at 2026-09-28 02:22:40 on dist-test-slave-h37r"
server_key: "c3a25b90fabc13fc5699f7cdadc432b5"
server_key_iv: "8bf0d61471662d4f6a0e5f0a9a4c6b19"
server_key_version: "encryptionkey@0"
I20260928 02:22:40.952847 25134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:22:40.956391 25149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:40.957827 25134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 02:22:40.958040 25134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "765a581f57854f509e8d4a818333e25d"
format_stamp: "Formatted at 2026-09-28 02:22:40 on dist-test-slave-h37r"
server_key: "c3a25b90fabc13fc5699f7cdadc432b5"
server_key_iv: "8bf0d61471662d4f6a0e5f0a9a4c6b19"
server_key_version: "encryptionkey@0"
I20260928 02:22:40.958237 25134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:40.977721 25134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:40.981992 25134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:40.982285 25134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:41.002553 25134 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42385
I20260928 02:22:41.002570 25200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42385 every 8 connection(s)
I20260928 02:22:41.004016 25134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:22:41.008212 25201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:41.011757 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 25134
I20260928 02:22:41.011965 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:22:41.012434 4659 external_mini_cluster.cc:1428] Setting key e98871bad09639d67cb3dde787ee189f
I20260928 02:22:41.021406 25201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d: Bootstrap starting.
I20260928 02:22:41.025517 25201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:41.025933 25204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:41.014383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54081 (local address 127.0.0.1:42385)
0928 02:22:41.015191 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:41.015204 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:41.015247 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:22:41.016981 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:41.017004 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:41.017442 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:41.017840 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:41.018393 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:41.019731 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:41.021114 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:41.021949 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:41.022142 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:41.023530 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:41.023538 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:41.023551 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:41.024180 (+ 629us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:41.024257 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:41.024282 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:41.024455 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:41.024720 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 02:22:41.024902 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":361,"threads_started":1}
I20260928 02:22:41.027082 25201 log.cc:824] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:41.030580 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:22:41.030654 25201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d: No bootstrap required, opened a new log
I20260928 02:22:41.042171 25201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "765a581f57854f509e8d4a818333e25d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42385 } }
I20260928 02:22:41.042829 25201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:41.042984 25201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 765a581f57854f509e8d4a818333e25d, State: Initialized, Role: FOLLOWER
I20260928 02:22:41.043170 25204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:41.035863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36951 (local address 127.0.0.1:42385)
0928 02:22:41.036114 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:41.036122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:41.036156 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:22:41.036260 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:41.036267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:41.036355 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:41.036502 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:41.036986 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:41.038398 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:41.039780 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:41.040657 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:41.040845 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:41.042084 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:41.042093 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:41.042099 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:41.042128 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:41.042190 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:41.042220 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:41.042378 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:41.042543 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 02:22:41.042607 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 02:22:41.044018 25201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "765a581f57854f509e8d4a818333e25d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42385 } }
I20260928 02:22:41.044342 25201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:22:41.044534 25201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:22:41.044787 25201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:41.047397 25201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "765a581f57854f509e8d4a818333e25d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42385 } }
I20260928 02:22:41.048097 25201 leader_election.cc:302] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 765a581f57854f509e8d4a818333e25d; no voters:
I20260928 02:22:41.048800 25201 leader_election.cc:288] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:22:41.048938 25211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:41.050316 25211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d [term 1 LEADER]: Becoming Leader. State: Replica: 765a581f57854f509e8d4a818333e25d, State: Running, Role: LEADER
I20260928 02:22:41.050982 25211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "765a581f57854f509e8d4a818333e25d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42385 } }
I20260928 02:22:41.051517 25201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:22:41.054752 25213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 765a581f57854f509e8d4a818333e25d. Latest consensus state: current_term: 1 leader_uuid: "765a581f57854f509e8d4a818333e25d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "765a581f57854f509e8d4a818333e25d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42385 } } }
I20260928 02:22:41.054801 25212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "765a581f57854f509e8d4a818333e25d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "765a581f57854f509e8d4a818333e25d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42385 } } }
I20260928 02:22:41.055490 25212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:41.055501 25213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:41.061373 25219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:22:41.069195 25219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:22:41.079536 25219 catalog_manager.cc:1347] Generated new cluster ID: e7bc10149d13487a91fa1d308c373caa
I20260928 02:22:41.079692 25219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:22:41.101907 25219 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:22:41.103268 25219 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:22:41.114959 25219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 765a581f57854f509e8d4a818333e25d: Generated new TSK 0
I20260928 02:22:41.116101 25219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:22:41.136121 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 25134
W20260928 02:22:41.137409 25196 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54081
W20260928 02:22:41.138247 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 25134: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:22:41.138482 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 25134
W20260928 02:22:41.156051 25119 connection.cc:535] client connection to 127.0.0.1:42385 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42385): Connection reset by peer (error 104)
2026-09-28T02:22:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1933 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:22:41 dist-test-slave-h37r krb5kdc[25232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:22:41 dist-test-slave-h37r krb5kdc[25232](info): set up 2 sockets
Sep 28 02:22:41 dist-test-slave-h37r krb5kdc[25232](info): commencing operation
krb5kdc: starting...
W20260928 02:22:42.252470 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:22:42 dist-test-slave-h37r krb5kdc[25232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:22:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:22:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:22:42.523345 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43331
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:42.886633 25245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:42.886953 25245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:42.887094 25245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:42.894533 25245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:22:42.894660 25245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:42.894707 25245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:42.894761 25245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:22:42.894807 25245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:22:42.905181 25245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43331
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:42.906534 25245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:42.907951 25245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:42.920547 25250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:42.920475 25253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:42.920636 25251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:42.921342 25245 server_base.cc:1034] running on GCE node
I20260928 02:22:42.922169 25245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:42.923547 25245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:42.924753 25245 hybrid_clock.cc:648] HybridClock initialized: now 1790562162924695 us; error 55 us; skew 500 ppm
I20260928 02:22:42.925129 25245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:42 dist-test-slave-h37r krb5kdc[25232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:42.928419 25245 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:22:42.929957 25245 webserver.cc:460] Webserver started at http://127.0.0.1:38253/ using document root <none> and password file <none>
I20260928 02:22:42.930615 25245 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:42.930717 25245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:42.930972 25245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:42.933992 25245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "d5a96ef00aa04e3aa4716587f5ff6f15"
format_stamp: "Formatted at 2026-09-28 02:22:42 on dist-test-slave-h37r"
server_key: "2f5ad4f789c18f77de0c4d11e4c1a652"
server_key_iv: "ddc285084e7cee31e6878f9ba6f05e1a"
server_key_version: "encryptionkey@0"
I20260928 02:22:42.934736 25245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "d5a96ef00aa04e3aa4716587f5ff6f15"
format_stamp: "Formatted at 2026-09-28 02:22:42 on dist-test-slave-h37r"
server_key: "2f5ad4f789c18f77de0c4d11e4c1a652"
server_key_iv: "ddc285084e7cee31e6878f9ba6f05e1a"
server_key_version: "encryptionkey@0"
I20260928 02:22:42.940083 25245 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:22:42.943601 25260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:42.945111 25245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:22:42.945255 25245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "d5a96ef00aa04e3aa4716587f5ff6f15"
format_stamp: "Formatted at 2026-09-28 02:22:42 on dist-test-slave-h37r"
server_key: "2f5ad4f789c18f77de0c4d11e4c1a652"
server_key_iv: "ddc285084e7cee31e6878f9ba6f05e1a"
server_key_version: "encryptionkey@0"
I20260928 02:22:42.945427 25245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:42.951424 25245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:42.955623 25245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:42.955878 25245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:42.975853 25245 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43331
I20260928 02:22:42.975950 25311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43331 every 8 connection(s)
I20260928 02:22:42.977337 25245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:22:42.979100 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 25245
I20260928 02:22:42.979282 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:22:42.979835 4659 external_mini_cluster.cc:1428] Setting key 0570fedda3eba55df426673bceeb8c78
I20260928 02:22:42.982129 25312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:42.987582 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:42.981647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45349 (local address 127.0.0.1:43331)
0928 02:22:42.982446 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:42.982458 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:42.982502 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:22:42.984143 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:42.984166 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:42.984505 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:42.984838 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:42.985951 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:42.985981 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:42.985993 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:42.986008 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:42.986094 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:42.986118 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:42.986287 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:42.986389 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 02:22:42.986588 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":347,"threads_started":1}
I20260928 02:22:42.991386 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:22:42.994796 25312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15: Bootstrap starting.
I20260928 02:22:42.998384 25312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:42.998649 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:42.995826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45591 (local address 127.0.0.1:43331)
0928 02:22:42.996015 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:42.996020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:42.996456 (+ 436us) server_negotiation.cc:407] Connection header received
0928 02:22:42.996661 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:42.996666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:42.996717 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:42.996831 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:42.997887 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:42.997896 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:42.997906 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:42.997923 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:42.997966 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:42.997983 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:42.998116 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:42.998256 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 02:22:42.998326 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:22:42.999744 25312 log.cc:824] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:43.002585 25312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15: No bootstrap required, opened a new log
I20260928 02:22:43.011404 25312 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a96ef00aa04e3aa4716587f5ff6f15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43331 } }
I20260928 02:22:43.011940 25312 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:43.012048 25312 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5a96ef00aa04e3aa4716587f5ff6f15, State: Initialized, Role: FOLLOWER
I20260928 02:22:43.012614 25312 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a96ef00aa04e3aa4716587f5ff6f15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43331 } }
I20260928 02:22:43.012823 25312 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:22:43.012934 25312 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:22:43.013063 25312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:43.015380 25312 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a96ef00aa04e3aa4716587f5ff6f15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43331 } }
I20260928 02:22:43.015852 25312 leader_election.cc:302] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5a96ef00aa04e3aa4716587f5ff6f15; no voters:
I20260928 02:22:43.016456 25312 leader_election.cc:288] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:22:43.016568 25322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:43.017438 25322 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15 [term 1 LEADER]: Becoming Leader. State: Replica: d5a96ef00aa04e3aa4716587f5ff6f15, State: Running, Role: LEADER
I20260928 02:22:43.018054 25322 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a96ef00aa04e3aa4716587f5ff6f15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43331 } }
I20260928 02:22:43.018946 25312 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:22:43.021376 25323 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5a96ef00aa04e3aa4716587f5ff6f15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a96ef00aa04e3aa4716587f5ff6f15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43331 } } }
I20260928 02:22:43.021364 25324 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5a96ef00aa04e3aa4716587f5ff6f15. Latest consensus state: current_term: 1 leader_uuid: "d5a96ef00aa04e3aa4716587f5ff6f15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a96ef00aa04e3aa4716587f5ff6f15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43331 } } }
I20260928 02:22:43.021869 25323 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:43.021934 25324 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:43.023396 25327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:22:43.028641 25327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:22:43.038067 25327 catalog_manager.cc:1347] Generated new cluster ID: d0adc2810c7e40198f8108b0f0dda394
I20260928 02:22:43.038197 25327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:22:43.084010 25327 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:22:43.084506 25327 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:22:43.099395 25327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5a96ef00aa04e3aa4716587f5ff6f15: Generated new TSK 0
I20260928 02:22:43.100454 25327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:22:43.115641 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 25245
W20260928 02:22:43.116796 25307 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45349
W20260928 02:22:43.117486 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 25245: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:22:43.117633 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 25245
2026-09-28T02:22:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1978 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:22:43 dist-test-slave-h37r krb5kdc[25343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:22:43 dist-test-slave-h37r krb5kdc[25343](info): set up 2 sockets
Sep 28 02:22:43 dist-test-slave-h37r krb5kdc[25343](info): commencing operation
krb5kdc: starting...
W20260928 02:22:44.228313 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:22:44 dist-test-slave-h37r krb5kdc[25343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:22:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:22:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:22:44.489614 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37045
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:44.823603 25356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:44.823971 25356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:44.824139 25356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:44.831553 25356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:22:44.831698 25356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:44.831795 25356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:44.831930 25356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:22:44.832012 25356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:22:44.841889 25356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:44.843220 25356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:44.844636 25356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:44.856513 25364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:44.856468 25361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:44.856554 25362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:44.857690 25356 server_base.cc:1034] running on GCE node
I20260928 02:22:44.858489 25356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:44.859804 25356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:44.861037 25356 hybrid_clock.cc:648] HybridClock initialized: now 1790562164860990 us; error 37 us; skew 500 ppm
I20260928 02:22:44.861387 25356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:44 dist-test-slave-h37r krb5kdc[25343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:44.877249 25356 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:22:44.879030 25356 webserver.cc:460] Webserver started at http://127.0.0.1:42313/ using document root <none> and password file <none>
I20260928 02:22:44.879801 25356 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:44.880013 25356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:44.880342 25356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:44.883437 25356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "9d93bccac5454bec80ae91692df0c208"
format_stamp: "Formatted at 2026-09-28 02:22:44 on dist-test-slave-h37r"
server_key: "08771baa4a0b4e5561361aad2441caee"
server_key_iv: "09b1f74d02cdb0cb8645871e13f68d95"
server_key_version: "encryptionkey@0"
I20260928 02:22:44.884351 25356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "9d93bccac5454bec80ae91692df0c208"
format_stamp: "Formatted at 2026-09-28 02:22:44 on dist-test-slave-h37r"
server_key: "08771baa4a0b4e5561361aad2441caee"
server_key_iv: "09b1f74d02cdb0cb8645871e13f68d95"
server_key_version: "encryptionkey@0"
I20260928 02:22:44.889884 25356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:22:44.893581 25371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:44.895025 25356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:22:44.895219 25356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "9d93bccac5454bec80ae91692df0c208"
format_stamp: "Formatted at 2026-09-28 02:22:44 on dist-test-slave-h37r"
server_key: "08771baa4a0b4e5561361aad2441caee"
server_key_iv: "09b1f74d02cdb0cb8645871e13f68d95"
server_key_version: "encryptionkey@0"
I20260928 02:22:44.895428 25356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:44.915805 25356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:44.919929 25356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:44.920264 25356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:44.939630 25356 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37045
I20260928 02:22:44.939649 25422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37045 every 8 connection(s)
I20260928 02:22:44.941103 25356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:22:44.944590 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 25356
I20260928 02:22:44.944774 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:22:44.945153 25423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:44.945281 4659 external_mini_cluster.cc:1428] Setting key 225d31806021647f4b1c30870e6be0c4
I20260928 02:22:44.960536 25423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208: Bootstrap starting.
Sep 28 02:22:44 dist-test-slave-h37r krb5kdc[25343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:22:44.964807 25423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:44.966198 25423 log.cc:824] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:44.969885 25423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208: No bootstrap required, opened a new log
I20260928 02:22:44.970583 25426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:44.947116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33093 (local address 127.0.0.1:37045)
0928 02:22:44.947944 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:44.947960 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:44.948011 (+ 51us) server_negotiation.cc:407] Connection header received
0928 02:22:44.950428 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:44.950459 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:44.951046 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:44.951537 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:44.952900 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:44.954938 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:44.956323 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:44.957441 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:44.957664 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:44.961692 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:44.961703 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:44.961722 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:44.961775 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:44.965779 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:44.966576 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:44.966584 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:44.966595 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:44.966838 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:44.967308 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:44.967311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:44.967313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:44.968046 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:44.968327 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:44.968730 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 02:22:44.969193 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":324,"threads_started":1}
I20260928 02:22:44.974545 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:22:44.978550 25423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d93bccac5454bec80ae91692df0c208" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37045 } }
I20260928 02:22:44.979015 25423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:44.979192 25423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d93bccac5454bec80ae91692df0c208, State: Initialized, Role: FOLLOWER
I20260928 02:22:44.979910 25423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d93bccac5454bec80ae91692df0c208" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37045 } }
I20260928 02:22:44.980242 25423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:22:44.980397 25423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:22:44.980575 25423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:44.983399 25423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d93bccac5454bec80ae91692df0c208" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37045 } }
I20260928 02:22:44.984016 25423 leader_election.cc:302] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d93bccac5454bec80ae91692df0c208; no voters:
I20260928 02:22:44.984696 25423 leader_election.cc:288] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:22:44.984805 25433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:44.985579 25433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208 [term 1 LEADER]: Becoming Leader. State: Replica: 9d93bccac5454bec80ae91692df0c208, State: Running, Role: LEADER
I20260928 02:22:44.986268 25433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d93bccac5454bec80ae91692df0c208" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37045 } }
I20260928 02:22:44.988490 25433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d93bccac5454bec80ae91692df0c208. Latest consensus state: current_term: 1 leader_uuid: "9d93bccac5454bec80ae91692df0c208" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d93bccac5454bec80ae91692df0c208" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37045 } } }
I20260928 02:22:44.988844 25433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:44.989494 25435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d93bccac5454bec80ae91692df0c208" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d93bccac5454bec80ae91692df0c208" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37045 } } }
I20260928 02:22:44.989995 25435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:44.993458 25423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:22:44.994027 25437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:22:44.996351 25426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:44.978913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56117 (local address 127.0.0.1:37045)
0928 02:22:44.979105 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:44.979113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:44.979645 (+ 532us) server_negotiation.cc:407] Connection header received
0928 02:22:44.979808 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:44.979813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:44.979924 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:44.980038 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:44.981631 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:44.982727 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:44.984063 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:44.984798 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:44.984925 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:44.988158 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:44.988164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:44.988168 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:44.988213 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:44.992787 (+ 4574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:44.993816 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:44.993822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:44.993826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:44.993986 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:44.994416 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:44.994423 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:44.994426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:44.994787 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:44.995145 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:44.995433 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 02:22:44.995664 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:22:45.001565 25437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:22:45.014789 25437 catalog_manager.cc:1347] Generated new cluster ID: 318eed5b30db46029009b92a84446296
I20260928 02:22:45.014915 25437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:22:45.029810 25437 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:22:45.031642 25437 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:22:45.039454 25437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d93bccac5454bec80ae91692df0c208: Generated new TSK 0
I20260928 02:22:45.040417 25437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:22:45.054366 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 25356
I20260928 02:22:45.055717 25418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:45.149189 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 25356
2026-09-28T02:22:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2031 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:22:45 dist-test-slave-h37r krb5kdc[25455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:22:45 dist-test-slave-h37r krb5kdc[25455](info): set up 2 sockets
Sep 28 02:22:45 dist-test-slave-h37r krb5kdc[25455](info): commencing operation
krb5kdc: starting...
W20260928 02:22:46.252434 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:22:46 dist-test-slave-h37r krb5kdc[25455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:22:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:22:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:22:46.511601 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39137
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:46.846841 25468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:46.847139 25468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:46.847230 25468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:46.854882 25468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:22:46.855019 25468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:46.855067 25468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:46.855134 25468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:22:46.855178 25468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:22:46.866451 25468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:46.867835 25468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:46.869321 25468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:46.881987 25473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:46.881955 25474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:46.881916 25476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:46.882948 25468 server_base.cc:1034] running on GCE node
I20260928 02:22:46.883728 25468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:46.885248 25468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:46.886474 25468 hybrid_clock.cc:648] HybridClock initialized: now 1790562166886433 us; error 35 us; skew 500 ppm
I20260928 02:22:46.886816 25468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:46 dist-test-slave-h37r krb5kdc[25455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:46.889907 25468 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:22:46.891433 25468 webserver.cc:460] Webserver started at http://127.0.0.1:40507/ using document root <none> and password file <none>
I20260928 02:22:46.892230 25468 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:46.892374 25468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:46.892685 25468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:46.896028 25468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "48b00afef8814ee58674985220487f00"
format_stamp: "Formatted at 2026-09-28 02:22:46 on dist-test-slave-h37r"
server_key: "a971eb8d19fe4dddb540b13d3e9f288e"
server_key_iv: "8d864b99ce5c58cd7ee571b6b9b879a0"
server_key_version: "encryptionkey@0"
I20260928 02:22:46.897017 25468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "48b00afef8814ee58674985220487f00"
format_stamp: "Formatted at 2026-09-28 02:22:46 on dist-test-slave-h37r"
server_key: "a971eb8d19fe4dddb540b13d3e9f288e"
server_key_iv: "8d864b99ce5c58cd7ee571b6b9b879a0"
server_key_version: "encryptionkey@0"
I20260928 02:22:46.902344 25468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:22:46.905787 25483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:46.907081 25468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:22:46.907264 25468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "48b00afef8814ee58674985220487f00"
format_stamp: "Formatted at 2026-09-28 02:22:46 on dist-test-slave-h37r"
server_key: "a971eb8d19fe4dddb540b13d3e9f288e"
server_key_iv: "8d864b99ce5c58cd7ee571b6b9b879a0"
server_key_version: "encryptionkey@0"
I20260928 02:22:46.907459 25468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:46.921008 25468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:46.924984 25468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:46.925261 25468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:46.944237 25468 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39137
I20260928 02:22:46.944239 25534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39137 every 8 connection(s)
I20260928 02:22:46.945717 25468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:22:46.946799 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 25468
I20260928 02:22:46.946983 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:22:46.947540 4659 external_mini_cluster.cc:1428] Setting key 835bc1a733d467f79f6a9b1714b502a4
I20260928 02:22:46.950359 25535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:46.962016 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:46.949526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57483 (local address 127.0.0.1:39137)
0928 02:22:46.950429 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:46.950449 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:46.950507 (+ 58us) server_negotiation.cc:407] Connection header received
0928 02:22:46.952446 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:46.952470 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:46.952893 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:46.953282 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:46.953860 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:46.955390 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:46.957308 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:46.958007 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:46.958145 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:46.959509 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:46.959520 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:46.959533 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:46.960183 (+ 650us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:46.960265 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:46.960290 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:46.960441 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:46.960684 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 02:22:46.960980 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":396,"threads_started":1}
I20260928 02:22:46.963126 25535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00: Bootstrap starting.
I20260928 02:22:46.965754 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:22:46.966542 25535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:46.967775 25535 log.cc:824] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:46.970824 25535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00: No bootstrap required, opened a new log
I20260928 02:22:46.977633 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:46.971027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33599 (local address 127.0.0.1:39137)
0928 02:22:46.971213 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:46.971219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:46.971575 (+ 356us) server_negotiation.cc:407] Connection header received
0928 02:22:46.971711 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:46.971717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:46.971789 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:46.971929 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:46.972473 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:46.973606 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:46.974931 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:46.975591 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:46.975730 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:46.976839 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:46.976846 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:46.976851 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:46.976877 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:46.976927 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:46.976952 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:46.977057 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:46.977171 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 02:22:46.977273 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:22:46.980676 25535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b00afef8814ee58674985220487f00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39137 } }
I20260928 02:22:46.981034 25535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:46.981107 25535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b00afef8814ee58674985220487f00, State: Initialized, Role: FOLLOWER
I20260928 02:22:46.981678 25535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b00afef8814ee58674985220487f00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39137 } }
I20260928 02:22:46.981880 25535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:22:46.981980 25535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:22:46.982105 25535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:46.984719 25535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b00afef8814ee58674985220487f00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39137 } }
I20260928 02:22:46.985172 25535 leader_election.cc:302] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48b00afef8814ee58674985220487f00; no voters:
I20260928 02:22:46.985744 25535 leader_election.cc:288] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:22:46.985918 25545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:46.986644 25545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00 [term 1 LEADER]: Becoming Leader. State: Replica: 48b00afef8814ee58674985220487f00, State: Running, Role: LEADER
I20260928 02:22:46.987236 25545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b00afef8814ee58674985220487f00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39137 } }
I20260928 02:22:46.987808 25535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:22:46.990166 25547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48b00afef8814ee58674985220487f00. Latest consensus state: current_term: 1 leader_uuid: "48b00afef8814ee58674985220487f00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b00afef8814ee58674985220487f00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39137 } } }
I20260928 02:22:46.990649 25547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:46.990233 25546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48b00afef8814ee58674985220487f00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b00afef8814ee58674985220487f00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39137 } } }
I20260928 02:22:46.991143 25546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00 [sys.catalog]: This master's current role is: LEADER
W20260928 02:22:47.001647 25561 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:22:47.001770 25561 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:22:47.001937 25553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:22:47.003393 25553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:22:47.012127 25553 catalog_manager.cc:1347] Generated new cluster ID: f279f429e50345c68a365fded30776b7
I20260928 02:22:47.012235 25553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:22:47.040427 25553 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:22:47.042407 25553 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:22:47.053598 25553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48b00afef8814ee58674985220487f00: Generated new TSK 0
I20260928 02:22:47.054515 25553 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:22:47.070606 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 25468
W20260928 02:22:47.071758 25530 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57483
W20260928 02:22:47.072510 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 25468: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:22:47.072676 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 25468
2026-09-28T02:22:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1923 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:22:47 dist-test-slave-h37r krb5kdc[25566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:22:47 dist-test-slave-h37r krb5kdc[25566](info): set up 2 sockets
Sep 28 02:22:47 dist-test-slave-h37r krb5kdc[25566](info): commencing operation
krb5kdc: starting...
W20260928 02:22:48.184372 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:22:48 dist-test-slave-h37r krb5kdc[25566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:22:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:22:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:22:48.449302 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40639
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:48.790793 25579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:48.791167 25579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:48.791332 25579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:48.798882 25579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:22:48.799046 25579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:48.799154 25579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:48.799256 25579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:22:48.799335 25579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:22:48.809620 25579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:48.810988 25579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:48.812351 25579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:48.824839 25584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:48.824791 25585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:48.824762 25587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:48.825688 25579 server_base.cc:1034] running on GCE node
I20260928 02:22:48.826452 25579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:48.827965 25579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:48.829175 25579 hybrid_clock.cc:648] HybridClock initialized: now 1790562168829114 us; error 50 us; skew 500 ppm
I20260928 02:22:48.829525 25579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:48 dist-test-slave-h37r krb5kdc[25566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:48.832653 25579 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:22:48.834080 25579 webserver.cc:460] Webserver started at http://127.0.0.1:44533/ using document root <none> and password file <none>
I20260928 02:22:48.834789 25579 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:48.834934 25579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:48.835273 25579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:48.838436 25579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "3562c2ae87a845ce82674de58a93a1c2"
format_stamp: "Formatted at 2026-09-28 02:22:48 on dist-test-slave-h37r"
server_key: "f37d8834f42b9d911d3fef34d9934402"
server_key_iv: "b94ad4a2e67ccb4f31dc15cd04c55b15"
server_key_version: "encryptionkey@0"
I20260928 02:22:48.839260 25579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "3562c2ae87a845ce82674de58a93a1c2"
format_stamp: "Formatted at 2026-09-28 02:22:48 on dist-test-slave-h37r"
server_key: "f37d8834f42b9d911d3fef34d9934402"
server_key_iv: "b94ad4a2e67ccb4f31dc15cd04c55b15"
server_key_version: "encryptionkey@0"
I20260928 02:22:48.845042 25579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:22:48.848702 25594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:48.850152 25579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:22:48.850340 25579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "3562c2ae87a845ce82674de58a93a1c2"
format_stamp: "Formatted at 2026-09-28 02:22:48 on dist-test-slave-h37r"
server_key: "f37d8834f42b9d911d3fef34d9934402"
server_key_iv: "b94ad4a2e67ccb4f31dc15cd04c55b15"
server_key_version: "encryptionkey@0"
I20260928 02:22:48.850560 25579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:48.856698 25579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:48.860934 25579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:48.861261 25579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:48.880514 25579 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40639
I20260928 02:22:48.880558 25645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40639 every 8 connection(s)
I20260928 02:22:48.882004 25579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:22:48.885269 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 25579
I20260928 02:22:48.885475 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:22:48.885905 4659 external_mini_cluster.cc:1428] Setting key d957a21ede01b7bb3715c51ef3b96e28
I20260928 02:22:48.886406 25646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:48.895067 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:48.887769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35817 (local address 127.0.0.1:40639)
0928 02:22:48.888756 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:48.888769 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:48.888813 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:22:48.891305 (+ 2492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:48.891332 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:48.891742 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:48.892186 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:48.893346 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:48.893353 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:48.893367 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:48.893382 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:48.893474 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:48.893496 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:48.893675 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:48.893789 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 02:22:48.894007 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":444,"threads_started":1}
I20260928 02:22:48.899219 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:22:48.901079 25646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2: Bootstrap starting.
I20260928 02:22:48.904675 25646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:48.905723 25646 log.cc:824] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:48.906209 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:48.903646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41203 (local address 127.0.0.1:40639)
0928 02:22:48.903834 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:48.903846 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:48.904227 (+ 381us) server_negotiation.cc:407] Connection header received
0928 02:22:48.904429 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:48.904433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:48.904490 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:48.904596 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:48.905572 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:48.905578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:48.905582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:48.905605 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:48.905646 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:48.905660 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:48.905783 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:48.905875 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 02:22:48.905934 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:22:48.908820 25646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2: No bootstrap required, opened a new log
I20260928 02:22:48.916531 25646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3562c2ae87a845ce82674de58a93a1c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } }
I20260928 02:22:48.916941 25646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:48.917057 25646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3562c2ae87a845ce82674de58a93a1c2, State: Initialized, Role: FOLLOWER
I20260928 02:22:48.917783 25646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3562c2ae87a845ce82674de58a93a1c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } }
I20260928 02:22:48.918035 25646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:22:48.918197 25646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:22:48.918388 25646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:48.920639 25646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3562c2ae87a845ce82674de58a93a1c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } }
I20260928 02:22:48.921147 25646 leader_election.cc:302] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3562c2ae87a845ce82674de58a93a1c2; no voters:
I20260928 02:22:48.921758 25646 leader_election.cc:288] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:22:48.921932 25656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:48.922632 25656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2 [term 1 LEADER]: Becoming Leader. State: Replica: 3562c2ae87a845ce82674de58a93a1c2, State: Running, Role: LEADER
I20260928 02:22:48.923187 25656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3562c2ae87a845ce82674de58a93a1c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } }
I20260928 02:22:48.923851 25646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:22:48.926180 25657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3562c2ae87a845ce82674de58a93a1c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3562c2ae87a845ce82674de58a93a1c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } } }
I20260928 02:22:48.926678 25657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:48.926921 25658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3562c2ae87a845ce82674de58a93a1c2. Latest consensus state: current_term: 1 leader_uuid: "3562c2ae87a845ce82674de58a93a1c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3562c2ae87a845ce82674de58a93a1c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } } }
I20260928 02:22:48.927150 25658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:48.930143 25666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:22:48.934669 25666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:22:48.943552 25666 catalog_manager.cc:1347] Generated new cluster ID: b4fb06c0e70d4d42b323f2a33c9fcd82
I20260928 02:22:48.943675 25666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:22:48.963490 25666 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:22:48.963961 25666 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:22:48.980988 25666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3562c2ae87a845ce82674de58a93a1c2: Generated new TSK 0
I20260928 02:22:48.981966 25666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:22:48.989148 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 25579
W20260928 02:22:48.990316 25641 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35817
W20260928 02:22:48.991071 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 25579: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:22:48.991221 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 25579
2026-09-28T02:22:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1918 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:22:49 dist-test-slave-h37r krb5kdc[25677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:22:49 dist-test-slave-h37r krb5kdc[25677](info): set up 2 sockets
Sep 28 02:22:49 dist-test-slave-h37r krb5kdc[25677](info): commencing operation
krb5kdc: starting...
W20260928 02:22:50.116492 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:22:50 dist-test-slave-h37r krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:22:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:22:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:22:50.384944 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44757
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:50.717579 25690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:50.717865 25690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:50.718025 25690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:50.725271 25690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:22:50.725391 25690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:50.725499 25690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:50.725595 25690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:22:50.725659 25690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:22:50.735445 25690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:50.736768 25690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:50.738075 25690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:50.749686 25695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:50.749718 25698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:50.749782 25696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:50.750703 25690 server_base.cc:1034] running on GCE node
I20260928 02:22:50.751535 25690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:50.752959 25690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:50.754155 25690 hybrid_clock.cc:648] HybridClock initialized: now 1790562170754113 us; error 32 us; skew 500 ppm
I20260928 02:22:50.754490 25690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:50 dist-test-slave-h37r krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:50.766939 25690 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:22:50.768774 25690 webserver.cc:460] Webserver started at http://127.0.0.1:45121/ using document root <none> and password file <none>
I20260928 02:22:50.769486 25690 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:50.769611 25690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:50.769927 25690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:50.773084 25690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "2e73e4ea533b40c9b25fae1937417069"
format_stamp: "Formatted at 2026-09-28 02:22:50 on dist-test-slave-h37r"
server_key: "3776314868f0d34f9f1e327ccd2cd059"
server_key_iv: "3381fd5bad9530578cc3fc1bb978d9f7"
server_key_version: "encryptionkey@0"
I20260928 02:22:50.773869 25690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "2e73e4ea533b40c9b25fae1937417069"
format_stamp: "Formatted at 2026-09-28 02:22:50 on dist-test-slave-h37r"
server_key: "3776314868f0d34f9f1e327ccd2cd059"
server_key_iv: "3381fd5bad9530578cc3fc1bb978d9f7"
server_key_version: "encryptionkey@0"
I20260928 02:22:50.779111 25690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:22:50.782800 25705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:50.784327 25690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:22:50.784519 25690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "2e73e4ea533b40c9b25fae1937417069"
format_stamp: "Formatted at 2026-09-28 02:22:50 on dist-test-slave-h37r"
server_key: "3776314868f0d34f9f1e327ccd2cd059"
server_key_iv: "3381fd5bad9530578cc3fc1bb978d9f7"
server_key_version: "encryptionkey@0"
I20260928 02:22:50.784725 25690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:50.803064 25690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:50.808259 25690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:50.808629 25690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:50.829891 25690 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44757
I20260928 02:22:50.829906 25756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44757 every 8 connection(s)
I20260928 02:22:50.831590 25690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:22:50.835215 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 25690
I20260928 02:22:50.835448 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:22:50.836082 4659 external_mini_cluster.cc:1428] Setting key 1d5c1b6242daf965b5341856e706fa73
I20260928 02:22:50.836688 25757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:50.849397 25757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069: Bootstrap starting.
Sep 28 02:22:50 dist-test-slave-h37r krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:22:50.852783 25757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:50.854030 25757 log.cc:824] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:50.856997 25757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069: No bootstrap required, opened a new log
I20260928 02:22:50.859504 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:50.838188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:58473 (local address 127.0.0.1:44757)
0928 02:22:50.839004 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:50.839016 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:50.839061 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:22:50.841669 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:50.841694 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:50.842142 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:50.842582 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:50.844095 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:50.845360 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:50.846663 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:50.847389 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:50.847603 (+ 214us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:50.851699 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:50.851707 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:50.851721 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:50.851757 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:50.855318 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:50.856157 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:50.856164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:50.856170 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:50.856324 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:50.856776 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:50.856779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:50.856781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:50.857329 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:50.857683 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:50.858184 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 02:22:50.858501 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":349,"threads_started":1}
I20260928 02:22:50.863603 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:22:50.865643 25757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e73e4ea533b40c9b25fae1937417069" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44757 } }
I20260928 02:22:50.866005 25757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:50.866119 25757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e73e4ea533b40c9b25fae1937417069, State: Initialized, Role: FOLLOWER
I20260928 02:22:50.866806 25757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e73e4ea533b40c9b25fae1937417069" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44757 } }
I20260928 02:22:50.867090 25757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:22:50.867239 25757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:22:50.867404 25757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:50.870049 25757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e73e4ea533b40c9b25fae1937417069" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44757 } }
I20260928 02:22:50.870563 25757 leader_election.cc:302] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e73e4ea533b40c9b25fae1937417069; no voters:
I20260928 02:22:50.871268 25757 leader_election.cc:288] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:22:50.871373 25767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:50.873138 25767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069 [term 1 LEADER]: Becoming Leader. State: Replica: 2e73e4ea533b40c9b25fae1937417069, State: Running, Role: LEADER
I20260928 02:22:50.873800 25757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:22:50.873799 25767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e73e4ea533b40c9b25fae1937417069" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44757 } }
I20260928 02:22:50.878594 25768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e73e4ea533b40c9b25fae1937417069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e73e4ea533b40c9b25fae1937417069" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44757 } } }
I20260928 02:22:50.879055 25768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:50.879607 25769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e73e4ea533b40c9b25fae1937417069. Latest consensus state: current_term: 1 leader_uuid: "2e73e4ea533b40c9b25fae1937417069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e73e4ea533b40c9b25fae1937417069" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44757 } } }
I20260928 02:22:50.880061 25769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:50.886408 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:50.868017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:58119 (local address 127.0.0.1:44757)
0928 02:22:50.868202 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:50.868208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:50.868651 (+ 443us) server_negotiation.cc:407] Connection header received
0928 02:22:50.868860 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:50.868866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:50.868947 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:50.869052 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:50.870435 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:50.871286 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:50.872920 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:50.873563 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:50.873702 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:50.876580 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:50.876587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:50.876591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:50.876633 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:50.880804 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:50.882134 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:50.882142 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:50.882149 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:50.882352 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:50.883089 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:50.883097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:50.883103 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:50.883671 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:50.885410 (+ 1739us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:50.885650 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 02:22:50.885890 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:22:50.890727 25776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:22:50.896416 25776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:22:50.906567 25776 catalog_manager.cc:1347] Generated new cluster ID: 6baa2bf8c876478f8a56b5a4ebf86598
I20260928 02:22:50.906692 25776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:22:50.923349 25776 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:22:50.924743 25776 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:22:50.930960 25776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e73e4ea533b40c9b25fae1937417069: Generated new TSK 0
I20260928 02:22:50.932050 25776 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:22:50.946416 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 25690
I20260928 02:22:50.948513 25752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:51.042496 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 25690
W20260928 02:22:51.060446 25675 connection.cc:535] client connection to 127.0.0.1:44757 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44757): Connection reset by peer (error 104)
2026-09-28T02:22:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2051 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:22:51 dist-test-slave-h37r krb5kdc[25789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:22:51 dist-test-slave-h37r krb5kdc[25789](info): set up 2 sockets
Sep 28 02:22:51 dist-test-slave-h37r krb5kdc[25789](info): commencing operation
krb5kdc: starting...
W20260928 02:22:52.160432 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:22:52 dist-test-slave-h37r krb5kdc[25789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:22:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:22:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:22:52.434934 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45967
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:52.790577 25802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:52.790931 25802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:52.791079 25802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:52.798512 25802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:22:52.798674 25802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:52.798776 25802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:52.798885 25802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:22:52.798962 25802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:22:52.809165 25802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:52.810506 25802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:52.811796 25802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:52.823567 25810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:52.823544 25807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:52.823642 25808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:52.825399 25802 server_base.cc:1034] running on GCE node
I20260928 02:22:52.826148 25802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:52.827509 25802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:52.828729 25802 hybrid_clock.cc:648] HybridClock initialized: now 1790562172828688 us; error 34 us; skew 500 ppm
I20260928 02:22:52.829066 25802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:52 dist-test-slave-h37r krb5kdc[25789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:52.832162 25802 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:22:52.833607 25802 webserver.cc:460] Webserver started at http://127.0.0.1:39539/ using document root <none> and password file <none>
I20260928 02:22:52.834231 25802 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:52.834338 25802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:52.834586 25802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:52.837491 25802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "10c3d7f673de4693817a0b7fb7df892b"
format_stamp: "Formatted at 2026-09-28 02:22:52 on dist-test-slave-h37r"
server_key: "939e00b7d159ff1558fd0bfcc20630a4"
server_key_iv: "c653257451f956893fa38daa76298df2"
server_key_version: "encryptionkey@0"
I20260928 02:22:52.838176 25802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "10c3d7f673de4693817a0b7fb7df892b"
format_stamp: "Formatted at 2026-09-28 02:22:52 on dist-test-slave-h37r"
server_key: "939e00b7d159ff1558fd0bfcc20630a4"
server_key_iv: "c653257451f956893fa38daa76298df2"
server_key_version: "encryptionkey@0"
I20260928 02:22:52.843252 25802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:22:52.846763 25817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:52.848173 25802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:22:52.848378 25802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "10c3d7f673de4693817a0b7fb7df892b"
format_stamp: "Formatted at 2026-09-28 02:22:52 on dist-test-slave-h37r"
server_key: "939e00b7d159ff1558fd0bfcc20630a4"
server_key_iv: "c653257451f956893fa38daa76298df2"
server_key_version: "encryptionkey@0"
I20260928 02:22:52.848572 25802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:52.864331 25802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:52.868896 25802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:52.869285 25802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:52.888502 25802 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45967
I20260928 02:22:52.888542 25868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45967 every 8 connection(s)
I20260928 02:22:52.889909 25802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:22:52.891122 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 25802
I20260928 02:22:52.891337 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:22:52.892028 4659 external_mini_cluster.cc:1428] Setting key b9b42a9dfb73d53f72d721d6e82c1a8e
I20260928 02:22:52.895365 25869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:52.905162 25872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:52.893990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:56615 (local address 127.0.0.1:45967)
0928 02:22:52.894756 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:52.894768 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:52.894812 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:22:52.896664 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:52.896687 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:52.897072 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:52.897460 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:52.898001 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:52.899320 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:52.900762 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:52.901466 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:52.901647 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:52.902889 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:52.902897 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:52.902912 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:52.903446 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:52.903532 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:52.903556 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:52.903737 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:52.904017 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 02:22:52.904185 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":359,"threads_started":1}
I20260928 02:22:52.908054 25869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b: Bootstrap starting.
I20260928 02:22:52.909274 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:22:52.911481 25869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:52.912725 25869 log.cc:824] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:52.915737 25869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b: No bootstrap required, opened a new log
I20260928 02:22:52.919643 25872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:52.913459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:52291 (local address 127.0.0.1:45967)
0928 02:22:52.913679 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:52.913686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:52.914193 (+ 507us) server_negotiation.cc:407] Connection header received
0928 02:22:52.914373 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:52.914378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:52.914447 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:52.914540 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:52.915077 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:52.915832 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:52.917154 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:52.917649 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:52.917790 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:52.918885 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:52.918891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:52.918895 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:52.918917 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:52.918972 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:52.918986 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:52.919100 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:52.919298 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 02:22:52.919348 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:22:52.924464 25869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c3d7f673de4693817a0b7fb7df892b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45967 } }
I20260928 02:22:52.924983 25869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:52.925112 25869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c3d7f673de4693817a0b7fb7df892b, State: Initialized, Role: FOLLOWER
I20260928 02:22:52.925788 25869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c3d7f673de4693817a0b7fb7df892b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45967 } }
I20260928 02:22:52.926009 25869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:22:52.926115 25869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:22:52.926250 25869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:52.928601 25869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c3d7f673de4693817a0b7fb7df892b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45967 } }
I20260928 02:22:52.929061 25869 leader_election.cc:302] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10c3d7f673de4693817a0b7fb7df892b; no voters:
I20260928 02:22:52.929600 25869 leader_election.cc:288] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:22:52.929738 25879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:52.930619 25879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b [term 1 LEADER]: Becoming Leader. State: Replica: 10c3d7f673de4693817a0b7fb7df892b, State: Running, Role: LEADER
I20260928 02:22:52.931195 25879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c3d7f673de4693817a0b7fb7df892b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45967 } }
I20260928 02:22:52.931934 25869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:22:52.934176 25880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10c3d7f673de4693817a0b7fb7df892b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c3d7f673de4693817a0b7fb7df892b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45967 } } }
I20260928 02:22:52.934551 25880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:52.935406 25881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10c3d7f673de4693817a0b7fb7df892b. Latest consensus state: current_term: 1 leader_uuid: "10c3d7f673de4693817a0b7fb7df892b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c3d7f673de4693817a0b7fb7df892b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45967 } } }
I20260928 02:22:52.935977 25881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:52.937683 25886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:22:52.944072 25886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:22:52.953660 25886 catalog_manager.cc:1347] Generated new cluster ID: 52887a97f69e4efbb16ef20b8686815d
I20260928 02:22:52.953789 25886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:22:52.965521 25886 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:22:52.967113 25886 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:22:52.988183 25886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10c3d7f673de4693817a0b7fb7df892b: Generated new TSK 0
I20260928 02:22:52.989142 25886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:22:53.005903 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 25802
W20260928 02:22:53.007478 25864 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.219:56615
W20260928 02:22:53.008330 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 25802: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:22:53.008520 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 25802
W20260928 02:22:53.022783 25787 connection.cc:535] client connection to 127.0.0.1:45967 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45967): Connection reset by peer (error 104)
2026-09-28T02:22:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1962 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:22:53 dist-test-slave-h37r krb5kdc[25900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:22:53 dist-test-slave-h37r krb5kdc[25900](info): set up 2 sockets
Sep 28 02:22:53 dist-test-slave-h37r krb5kdc[25900](info): commencing operation
krb5kdc: starting...
W20260928 02:22:54.136487 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:22:54 dist-test-slave-h37r krb5kdc[25900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:22:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:22:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:22:54.405411 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45165
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:54.736894 25913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:54.737252 25913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:54.737432 25913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:54.744946 25913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:22:54.745100 25913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:54.745211 25913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:54.745319 25913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:22:54.745397 25913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:22:54.755620 25913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:54.757555 25913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:54.759110 25913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:54.771540 25921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:54.771492 25919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:54.771579 25918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:54.772531 25913 server_base.cc:1034] running on GCE node
I20260928 02:22:54.773267 25913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:54.774621 25913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:54.775857 25913 hybrid_clock.cc:648] HybridClock initialized: now 1790562174775807 us; error 32 us; skew 500 ppm
I20260928 02:22:54.776245 25913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:54 dist-test-slave-h37r krb5kdc[25900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:54.779508 25913 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:22:54.781138 25913 webserver.cc:460] Webserver started at http://127.0.0.1:40285/ using document root <none> and password file <none>
I20260928 02:22:54.781834 25913 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:54.781937 25913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:54.782200 25913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:54.785275 25913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "0d808341db614154b05b9cd9cbbbc372"
format_stamp: "Formatted at 2026-09-28 02:22:54 on dist-test-slave-h37r"
server_key: "a0e07bf98ca1c60102835ca80be94e00"
server_key_iv: "5d6576892449eaebc0c47fae19037a60"
server_key_version: "encryptionkey@0"
I20260928 02:22:54.785992 25913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "0d808341db614154b05b9cd9cbbbc372"
format_stamp: "Formatted at 2026-09-28 02:22:54 on dist-test-slave-h37r"
server_key: "a0e07bf98ca1c60102835ca80be94e00"
server_key_iv: "5d6576892449eaebc0c47fae19037a60"
server_key_version: "encryptionkey@0"
I20260928 02:22:54.791128 25913 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:22:54.794579 25928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:54.795881 25913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:22:54.796121 25913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "0d808341db614154b05b9cd9cbbbc372"
format_stamp: "Formatted at 2026-09-28 02:22:54 on dist-test-slave-h37r"
server_key: "a0e07bf98ca1c60102835ca80be94e00"
server_key_iv: "5d6576892449eaebc0c47fae19037a60"
server_key_version: "encryptionkey@0"
I20260928 02:22:54.796314 25913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:54.802014 25913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:54.806180 25913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:54.806490 25913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:54.826047 25913 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45165
I20260928 02:22:54.826087 25979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45165 every 8 connection(s)
I20260928 02:22:54.827528 25913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:22:54.831014 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 25913
I20260928 02:22:54.831218 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:22:54.831648 4659 external_mini_cluster.cc:1428] Setting key 8aca51d3a68bec2b28a9768221c3642a
I20260928 02:22:54.831784 25980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:54.840384 25983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:54.833526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:43647 (local address 127.0.0.1:45165)
0928 02:22:54.834308 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:54.834320 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:54.834367 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:22:54.836698 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:54.836721 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:54.837080 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:54.837463 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:54.838648 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:54.838655 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:54.838668 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:54.838689 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:54.838750 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:54.838773 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:54.838953 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:54.839075 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 02:22:54.839294 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":353,"threads_started":1}
I20260928 02:22:54.844269 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:22:54.846632 25980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372: Bootstrap starting.
I20260928 02:22:54.850457 25980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:54.851261 25983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:54.848559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:45361 (local address 127.0.0.1:45165)
0928 02:22:54.848751 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:54.848758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:54.849205 (+ 447us) server_negotiation.cc:407] Connection header received
0928 02:22:54.849337 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:54.849343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:54.849414 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:54.849548 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:54.850454 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:54.850461 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:54.850465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:54.850486 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:54.850540 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:54.850564 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:54.850702 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:54.850813 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 02:22:54.850890 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:22:54.851884 25980 log.cc:824] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:54.854889 25980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372: No bootstrap required, opened a new log
I20260928 02:22:54.862769 25980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d808341db614154b05b9cd9cbbbc372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45165 } }
I20260928 02:22:54.863193 25980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:54.863319 25980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d808341db614154b05b9cd9cbbbc372, State: Initialized, Role: FOLLOWER
I20260928 02:22:54.863974 25980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d808341db614154b05b9cd9cbbbc372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45165 } }
I20260928 02:22:54.864239 25980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:22:54.864401 25980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:22:54.864579 25980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:54.866885 25980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d808341db614154b05b9cd9cbbbc372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45165 } }
I20260928 02:22:54.867439 25980 leader_election.cc:302] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d808341db614154b05b9cd9cbbbc372; no voters:
I20260928 02:22:54.868134 25980 leader_election.cc:288] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:22:54.868287 25990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:54.869023 25990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372 [term 1 LEADER]: Becoming Leader. State: Replica: 0d808341db614154b05b9cd9cbbbc372, State: Running, Role: LEADER
I20260928 02:22:54.869611 25990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d808341db614154b05b9cd9cbbbc372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45165 } }
I20260928 02:22:54.870469 25980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:22:54.872653 25991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d808341db614154b05b9cd9cbbbc372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d808341db614154b05b9cd9cbbbc372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45165 } } }
I20260928 02:22:54.873180 25991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:54.873831 25992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d808341db614154b05b9cd9cbbbc372. Latest consensus state: current_term: 1 leader_uuid: "0d808341db614154b05b9cd9cbbbc372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d808341db614154b05b9cd9cbbbc372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45165 } } }
I20260928 02:22:54.874266 25992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:54.876179 25998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:22:54.880878 25998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:22:54.889330 25998 catalog_manager.cc:1347] Generated new cluster ID: de83178a481c4a3e8261ebdee2d33a5c
I20260928 02:22:54.889468 25998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:22:54.917549 25998 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:22:54.917996 25998 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:22:54.925100 25998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d808341db614154b05b9cd9cbbbc372: Generated new TSK 0
I20260928 02:22:54.925880 25998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:22:54.933491 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 25913
W20260928 02:22:54.934695 25975 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.219:43647
W20260928 02:22:54.935417 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 25913: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:22:54.935547 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 25913
2026-09-28T02:22:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1929 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:22:55 dist-test-slave-h37r krb5kdc[26011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:22:55 dist-test-slave-h37r krb5kdc[26011](info): set up 2 sockets
Sep 28 02:22:55 dist-test-slave-h37r krb5kdc[26011](info): commencing operation
krb5kdc: starting...
W20260928 02:22:56.064739 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.009s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:22:56 dist-test-slave-h37r krb5kdc[26011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:22:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:22:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:22:56.339834 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:56.682700 26024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:56.683064 26024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:56.683224 26024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:56.690793 26024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:22:56.690966 26024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:56.691068 26024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:56.691175 26024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:22:56.691259 26024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:22:56.702436 26024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45845
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:56.704061 26024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:56.705423 26024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:56.717564 26029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:56.717494 26030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:56.717526 26032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:56.718104 26024 server_base.cc:1034] running on GCE node
I20260928 02:22:56.718983 26024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:56.720386 26024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:56.721657 26024 hybrid_clock.cc:648] HybridClock initialized: now 1790562176721600 us; error 26 us; skew 500 ppm
I20260928 02:22:56.722082 26024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:56 dist-test-slave-h37r krb5kdc[26011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:56.733870 26024 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:22:56.735562 26024 webserver.cc:460] Webserver started at http://127.0.0.1:35235/ using document root <none> and password file <none>
I20260928 02:22:56.736410 26024 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:56.736553 26024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:56.736908 26024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:56.740067 26024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "58435abbc6a64c9298e7cfbfb395d3ba"
format_stamp: "Formatted at 2026-09-28 02:22:56 on dist-test-slave-h37r"
server_key: "0f46d166888affdb030b7581415e0c36"
server_key_iv: "747a42adb2f7b502530f3d1e658d21e7"
server_key_version: "encryptionkey@0"
I20260928 02:22:56.740850 26024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "58435abbc6a64c9298e7cfbfb395d3ba"
format_stamp: "Formatted at 2026-09-28 02:22:56 on dist-test-slave-h37r"
server_key: "0f46d166888affdb030b7581415e0c36"
server_key_iv: "747a42adb2f7b502530f3d1e658d21e7"
server_key_version: "encryptionkey@0"
I20260928 02:22:56.746137 26024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:22:56.749727 26039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:56.751065 26024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:22:56.751255 26024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "58435abbc6a64c9298e7cfbfb395d3ba"
format_stamp: "Formatted at 2026-09-28 02:22:56 on dist-test-slave-h37r"
server_key: "0f46d166888affdb030b7581415e0c36"
server_key_iv: "747a42adb2f7b502530f3d1e658d21e7"
server_key_version: "encryptionkey@0"
I20260928 02:22:56.751456 26024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:56.774070 26024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:56.778539 26024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:56.778855 26024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:56.797727 26024 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45845
I20260928 02:22:56.797750 26090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45845 every 8 connection(s)
I20260928 02:22:56.799188 26024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:22:56.802919 26091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:56.806178 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 26024
I20260928 02:22:56.806362 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:22:56.806900 4659 external_mini_cluster.cc:1428] Setting key 256cfb4ca2a0d5f129215fab6b74261c
I20260928 02:22:56.814626 26091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba: Bootstrap starting.
I20260928 02:22:56.817718 26091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:56.818712 26091 log.cc:824] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:22:56 dist-test-slave-h37r krb5kdc[26011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:22:56.821820 26091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba: No bootstrap required, opened a new log
I20260928 02:22:56.829959 26094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:56.808840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:46393 (local address 127.0.0.1:45845)
0928 02:22:56.809577 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:56.809589 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:56.809629 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:22:56.811318 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:56.811342 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:56.811780 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:56.812268 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:56.813746 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:56.815128 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:56.816486 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:56.817155 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:56.817360 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:56.822053 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:56.822060 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:56.822074 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:56.822109 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:56.825650 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:56.826519 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:56.826525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:56.826531 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:56.826707 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:56.827221 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:56.827225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:56.827230 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:56.827760 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:56.828139 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:56.828667 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 02:22:56.828975 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":305,"threads_started":1}
I20260928 02:22:56.830112 26091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58435abbc6a64c9298e7cfbfb395d3ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45845 } }
I20260928 02:22:56.830600 26091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:56.830720 26091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58435abbc6a64c9298e7cfbfb395d3ba, State: Initialized, Role: FOLLOWER
I20260928 02:22:56.831411 26091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58435abbc6a64c9298e7cfbfb395d3ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45845 } }
I20260928 02:22:56.831667 26091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:22:56.831854 26091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:22:56.832046 26091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:56.834378 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:22:56.834867 26091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58435abbc6a64c9298e7cfbfb395d3ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45845 } }
I20260928 02:22:56.835407 26091 leader_election.cc:302] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58435abbc6a64c9298e7cfbfb395d3ba; no voters:
I20260928 02:22:56.836124 26091 leader_election.cc:288] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:22:56.836216 26096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:56.838688 26091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:22:56.842926 26096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba [term 1 LEADER]: Becoming Leader. State: Replica: 58435abbc6a64c9298e7cfbfb395d3ba, State: Running, Role: LEADER
I20260928 02:22:56.843619 26096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58435abbc6a64c9298e7cfbfb395d3ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45845 } }
I20260928 02:22:56.849113 26097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58435abbc6a64c9298e7cfbfb395d3ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58435abbc6a64c9298e7cfbfb395d3ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45845 } } }
I20260928 02:22:56.849535 26097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:56.850011 26098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58435abbc6a64c9298e7cfbfb395d3ba. Latest consensus state: current_term: 1 leader_uuid: "58435abbc6a64c9298e7cfbfb395d3ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58435abbc6a64c9298e7cfbfb395d3ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45845 } } }
I20260928 02:22:56.850298 26098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:56.860790 26107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:22:56.865459 26094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:56.842218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:57441 (local address 127.0.0.1:45845)
0928 02:22:56.842443 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:56.842453 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:56.846593 (+ 4140us) server_negotiation.cc:407] Connection header received
0928 02:22:56.846827 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:56.846832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:56.846970 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:56.847118 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:56.848865 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:56.850242 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:56.852508 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:56.853419 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:56.853586 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:56.857555 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:56.857563 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:56.857567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:22:56.857608 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:56.861964 (+ 4356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:56.862856 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:56.862860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:56.862862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:56.863025 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:22:56.863560 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:22:56.863571 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:22:56.863574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:22:56.864080 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:56.864359 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:56.864814 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 02:22:56.865008 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:22:56.866488 26107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:22:56.875026 26107 catalog_manager.cc:1347] Generated new cluster ID: 68bd18ce81c34e25b477e9bf2592b43e
I20260928 02:22:56.875147 26107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:22:56.908934 26107 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:22:56.910312 26107 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:22:56.916738 26107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58435abbc6a64c9298e7cfbfb395d3ba: Generated new TSK 0
I20260928 02:22:56.917598 26107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:22:56.931625 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 26024
I20260928 02:22:56.933075 26086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:22:57.031253 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 26024
W20260928 02:22:57.049557 26009 connection.cc:535] client connection to 127.0.0.1:45845 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45845): Connection reset by peer (error 104)
2026-09-28T02:22:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2097 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:22:57 dist-test-slave-h37r krb5kdc[26123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:22:57 dist-test-slave-h37r krb5kdc[26123](info): set up 2 sockets
Sep 28 02:22:57 dist-test-slave-h37r krb5kdc[26123](info): commencing operation
krb5kdc: starting...
W20260928 02:22:58.156977 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:22:58 dist-test-slave-h37r krb5kdc[26123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:22:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:22:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:22:58.436134 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37749
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:22:58.776108 26136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:22:58.776479 26136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:22:58.776640 26136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:22:58.784302 26136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:22:58.784473 26136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:22:58.784582 26136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:22:58.784693 26136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:22:58.784775 26136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:22:58.795231 26136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37749
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:22:58.796731 26136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:22:58.798139 26136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:22:58.810290 26141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:58.811020 26142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:22:58.811029 26144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:22:58.812497 26136 server_base.cc:1034] running on GCE node
I20260928 02:22:58.813365 26136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:22:58.814944 26136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:22:58.816187 26136 hybrid_clock.cc:648] HybridClock initialized: now 1790562178816145 us; error 66 us; skew 500 ppm
I20260928 02:22:58.816603 26136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:22:58 dist-test-slave-h37r krb5kdc[26123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:22:58.820055 26136 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:22:58.821708 26136 webserver.cc:460] Webserver started at http://127.0.0.1:37271/ using document root <none> and password file <none>
I20260928 02:22:58.822439 26136 fs_manager.cc:359] Metadata directory not provided
I20260928 02:22:58.822592 26136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:22:58.822909 26136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:22:58.826479 26136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "bb43f97d358244598e9e1758f9c0fae2"
format_stamp: "Formatted at 2026-09-28 02:22:58 on dist-test-slave-h37r"
server_key: "b10f252053faff5d347fb5f004224d82"
server_key_iv: "15ffdc20ce57f8fc91fde97d43cc8082"
server_key_version: "encryptionkey@0"
I20260928 02:22:58.827265 26136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "bb43f97d358244598e9e1758f9c0fae2"
format_stamp: "Formatted at 2026-09-28 02:22:58 on dist-test-slave-h37r"
server_key: "b10f252053faff5d347fb5f004224d82"
server_key_iv: "15ffdc20ce57f8fc91fde97d43cc8082"
server_key_version: "encryptionkey@0"
I20260928 02:22:58.832710 26136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:22:58.836608 26151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:22:58.838094 26136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:22:58.838302 26136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "bb43f97d358244598e9e1758f9c0fae2"
format_stamp: "Formatted at 2026-09-28 02:22:58 on dist-test-slave-h37r"
server_key: "b10f252053faff5d347fb5f004224d82"
server_key_iv: "15ffdc20ce57f8fc91fde97d43cc8082"
server_key_version: "encryptionkey@0"
I20260928 02:22:58.838557 26136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:22:58.861213 26136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:22:58.865900 26136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:22:58.866217 26136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:22:58.886317 26136 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37749
I20260928 02:22:58.886363 26202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37749 every 8 connection(s)
I20260928 02:22:58.887837 26136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:22:58.892089 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 26136
I20260928 02:22:58.892305 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:22:58.892289 26203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:22:58.892925 4659 external_mini_cluster.cc:1428] Setting key 9b250f0a79d0d5771e559fda2e0867a8
I20260928 02:22:58.906441 26203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2: Bootstrap starting.
I20260928 02:22:58.906960 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:58.894962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:37239 (local address 127.0.0.1:37749)
0928 02:22:58.895866 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:58.895885 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:58.895984 (+ 99us) server_negotiation.cc:407] Connection header received
0928 02:22:58.897960 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:58.897990 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:58.898451 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:58.898896 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:58.899503 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:58.900894 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:58.902272 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:58.902974 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:58.903207 (+ 233us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:58.904598 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:58.904605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:58.904619 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:58.905170 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:58.905254 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:58.905279 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:58.905459 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:58.905721 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 02:22:58.905907 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":350,"threads_started":1}
I20260928 02:22:58.909854 26203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2: Neither blocks nor log segments found. Creating new log.
I20260928 02:22:58.911000 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:22:58.911129 26203 log.cc:824] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2: Log is configured to *not* fsync() on all Append() calls
I20260928 02:22:58.914762 26203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2: No bootstrap required, opened a new log
I20260928 02:22:58.922662 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:22:58.916116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:58143 (local address 127.0.0.1:37749)
0928 02:22:58.916277 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 02:22:58.916282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:22:58.916835 (+ 553us) server_negotiation.cc:407] Connection header received
0928 02:22:58.917054 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:22:58.917059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:22:58.917135 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:22:58.917246 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:22:58.917778 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:58.918590 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:58.919990 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:22:58.920483 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:22:58.920672 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:22:58.921873 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:22:58.921877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:22:58.921881 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:22:58.921899 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:22:58.921952 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:22:58.921971 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:22:58.922108 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 02:22:58.922297 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 02:22:58.922356 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:22:58.923959 26203 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb43f97d358244598e9e1758f9c0fae2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37749 } }
I20260928 02:22:58.924542 26203 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:22:58.924676 26203 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb43f97d358244598e9e1758f9c0fae2, State: Initialized, Role: FOLLOWER
I20260928 02:22:58.925333 26203 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb43f97d358244598e9e1758f9c0fae2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37749 } }
I20260928 02:22:58.925639 26203 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:22:58.925827 26203 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:22:58.926110 26203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:22:58.928915 26203 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb43f97d358244598e9e1758f9c0fae2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37749 } }
I20260928 02:22:58.929600 26203 leader_election.cc:302] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb43f97d358244598e9e1758f9c0fae2; no voters:
I20260928 02:22:58.930332 26203 leader_election.cc:288] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:22:58.930466 26213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:22:58.931130 26213 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2 [term 1 LEADER]: Becoming Leader. State: Replica: bb43f97d358244598e9e1758f9c0fae2, State: Running, Role: LEADER
I20260928 02:22:58.931718 26213 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb43f97d358244598e9e1758f9c0fae2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37749 } }
I20260928 02:22:58.933099 26203 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:22:58.934913 26214 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb43f97d358244598e9e1758f9c0fae2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb43f97d358244598e9e1758f9c0fae2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37749 } } }
I20260928 02:22:58.934885 26215 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb43f97d358244598e9e1758f9c0fae2. Latest consensus state: current_term: 1 leader_uuid: "bb43f97d358244598e9e1758f9c0fae2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb43f97d358244598e9e1758f9c0fae2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37749 } } }
I20260928 02:22:58.935369 26215 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:58.935369 26214 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2 [sys.catalog]: This master's current role is: LEADER
I20260928 02:22:58.938136 26218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:22:58.944798 26218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:22:58.951805 26230 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:22:58.956889 26218 catalog_manager.cc:1347] Generated new cluster ID: 157f667432d640ef925d5bc87a9aa663
I20260928 02:22:58.957032 26218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:22:58.965775 26218 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:22:58.967279 26218 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:22:58.978884 26218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb43f97d358244598e9e1758f9c0fae2: Generated new TSK 0
I20260928 02:22:58.980115 26218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:22:58.996433 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 26136
W20260928 02:22:58.997637 26198 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.219:37239
W20260928 02:22:58.998437 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 26136: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:22:58.998588 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 26136
W20260928 02:22:59.015699 26121 connection.cc:535] client connection to 127.0.0.1:37749 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37749): Connection reset by peer (error 104)
2026-09-28T02:22:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1966 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:22:59 dist-test-slave-h37r krb5kdc[26234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:22:59 dist-test-slave-h37r krb5kdc[26234](info): set up 2 sockets
Sep 28 02:22:59 dist-test-slave-h37r krb5kdc[26234](info): commencing operation
krb5kdc: starting...
W20260928 02:23:00.140566 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:23:00 dist-test-slave-h37r krb5kdc[26234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:23:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:23:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:23:00.413261 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44233
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:23:00.744828 26247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:23:00.745124 26247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:23:00.745321 26247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:23:00.752753 26247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:23:00.752902 26247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:23:00.753010 26247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:23:00.753116 26247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:23:00.753196 26247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:23:00.766472 26247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44233
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:23:00.767849 26247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:23:00.769301 26247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:23:00.781571 26252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:00.781601 26255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:00.781654 26253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:23:00.782961 26247 server_base.cc:1034] running on GCE node
I20260928 02:23:00.783838 26247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:23:00.785404 26247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:23:00.786623 26247 hybrid_clock.cc:648] HybridClock initialized: now 1790562180786566 us; error 39 us; skew 500 ppm
I20260928 02:23:00.787141 26247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:23:00 dist-test-slave-h37r krb5kdc[26234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:23:00.790679 26247 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:23:00.792387 26247 webserver.cc:460] Webserver started at http://127.0.0.1:39231/ using document root <none> and password file <none>
I20260928 02:23:00.793123 26247 fs_manager.cc:359] Metadata directory not provided
I20260928 02:23:00.793284 26247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:23:00.793602 26247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:23:00.796763 26247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "0bd24e07c0c84045bbacf4049d670c82"
format_stamp: "Formatted at 2026-09-28 02:23:00 on dist-test-slave-h37r"
server_key: "a6b904b1ac9597b5505644f782d6b5c1"
server_key_iv: "a4aff7efcecad8451cdc89b4170aa259"
server_key_version: "encryptionkey@0"
I20260928 02:23:00.797537 26247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "0bd24e07c0c84045bbacf4049d670c82"
format_stamp: "Formatted at 2026-09-28 02:23:00 on dist-test-slave-h37r"
server_key: "a6b904b1ac9597b5505644f782d6b5c1"
server_key_iv: "a4aff7efcecad8451cdc89b4170aa259"
server_key_version: "encryptionkey@0"
I20260928 02:23:00.802834 26247 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 02:23:00.806427 26262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:23:00.807866 26247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:23:00.808122 26247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "0bd24e07c0c84045bbacf4049d670c82"
format_stamp: "Formatted at 2026-09-28 02:23:00 on dist-test-slave-h37r"
server_key: "a6b904b1ac9597b5505644f782d6b5c1"
server_key_iv: "a4aff7efcecad8451cdc89b4170aa259"
server_key_version: "encryptionkey@0"
I20260928 02:23:00.808353 26247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:23:00.814293 26247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:23:00.818845 26247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:23:00.819155 26247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:23:00.838356 26247 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44233
I20260928 02:23:00.838393 26313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44233 every 8 connection(s)
I20260928 02:23:00.839846 26247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:23:00.843678 26314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:23:00.849011 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 26247
I20260928 02:23:00.849200 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:23:00.849730 4659 external_mini_cluster.cc:1428] Setting key 8c932e9b86bfbd9f7a7c6edda8fc9feb
I20260928 02:23:00.855104 26314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82: Bootstrap starting.
I20260928 02:23:00.857892 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:00.851712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:39643 (local address 127.0.0.1:44233)
0928 02:23:00.852645 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:00.852658 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:00.852700 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:23:00.854445 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:00.854469 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:00.854837 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:00.855183 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:00.856264 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:00.856270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:00.856282 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:23:00.856296 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:00.856405 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:23:00.856425 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:00.856593 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:00.856716 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 02:23:00.856947 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":460,"threads_started":1}
I20260928 02:23:00.858718 26314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82: Neither blocks nor log segments found. Creating new log.
I20260928 02:23:00.859800 26314 log.cc:824] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82: Log is configured to *not* fsync() on all Append() calls
I20260928 02:23:00.861660 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:23:00.863170 26314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82: No bootstrap required, opened a new log
I20260928 02:23:00.869249 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:00.866697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:47095 (local address 127.0.0.1:44233)
0928 02:23:00.866885 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:00.866891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:00.867353 (+ 462us) server_negotiation.cc:407] Connection header received
0928 02:23:00.867476 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:00.867482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:00.867555 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:00.867671 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:00.868599 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:00.868605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:00.868610 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:23:00.868628 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:00.868679 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:23:00.868704 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:00.868817 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:00.868902 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 02:23:00.868963 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:23:00.872639 26314 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd24e07c0c84045bbacf4049d670c82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44233 } }
I20260928 02:23:00.873003 26314 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:23:00.873076 26314 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bd24e07c0c84045bbacf4049d670c82, State: Initialized, Role: FOLLOWER
I20260928 02:23:00.873634 26314 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd24e07c0c84045bbacf4049d670c82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44233 } }
I20260928 02:23:00.873910 26314 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:23:00.874058 26314 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:23:00.874267 26314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:23:00.876711 26314 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd24e07c0c84045bbacf4049d670c82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44233 } }
I20260928 02:23:00.877228 26314 leader_election.cc:302] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bd24e07c0c84045bbacf4049d670c82; no voters:
I20260928 02:23:00.877832 26314 leader_election.cc:288] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:23:00.877987 26324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:23:00.878661 26324 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82 [term 1 LEADER]: Becoming Leader. State: Replica: 0bd24e07c0c84045bbacf4049d670c82, State: Running, Role: LEADER
I20260928 02:23:00.879212 26324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd24e07c0c84045bbacf4049d670c82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44233 } }
I20260928 02:23:00.880069 26314 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:23:00.882494 26325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bd24e07c0c84045bbacf4049d670c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd24e07c0c84045bbacf4049d670c82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44233 } } }
I20260928 02:23:00.882537 26326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bd24e07c0c84045bbacf4049d670c82. Latest consensus state: current_term: 1 leader_uuid: "0bd24e07c0c84045bbacf4049d670c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd24e07c0c84045bbacf4049d670c82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44233 } } }
I20260928 02:23:00.883033 26325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82 [sys.catalog]: This master's current role is: LEADER
I20260928 02:23:00.883082 26326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82 [sys.catalog]: This master's current role is: LEADER
I20260928 02:23:00.883838 26333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:23:00.889858 26333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:23:00.899626 26333 catalog_manager.cc:1347] Generated new cluster ID: f6681deab51443f1b0090ef000321a6e
I20260928 02:23:00.899755 26333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:23:00.945925 26333 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:23:00.946528 26333 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:23:00.963424 26333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bd24e07c0c84045bbacf4049d670c82: Generated new TSK 0
I20260928 02:23:00.964604 26333 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:23:00.979096 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 26247
W20260928 02:23:00.980327 26309 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.219:39643
W20260928 02:23:00.981092 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 26247: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:23:00.981279 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 26247
2026-09-28T02:23:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1983 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:23:01 dist-test-slave-h37r krb5kdc[26345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:23:01 dist-test-slave-h37r krb5kdc[26345](info): set up 2 sockets
Sep 28 02:23:01 dist-test-slave-h37r krb5kdc[26345](info): commencing operation
krb5kdc: starting...
W20260928 02:23:02.104468 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:23:02 dist-test-slave-h37r krb5kdc[26345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:23:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:23:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:23:02.369484 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:43405
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:43405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:23:02.721592 26358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:23:02.721943 26358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:23:02.722110 26358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:23:02.729589 26358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:23:02.729753 26358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:23:02.729859 26358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:23:02.729966 26358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:23:02.730057 26358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:23:02.740420 26358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:43405
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:43405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:23:02.741812 26358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:23:02.743158 26358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:23:02.755410 26366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:02.755477 26363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:02.755504 26364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:23:02.756170 26358 server_base.cc:1034] running on GCE node
I20260928 02:23:02.757193 26358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:23:02.759162 26358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:23:02.760386 26358 hybrid_clock.cc:648] HybridClock initialized: now 1790562182760328 us; error 46 us; skew 500 ppm
I20260928 02:23:02.760856 26358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:23:02 dist-test-slave-h37r krb5kdc[26345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562182, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:23:02.765161 26358 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:23:02.766862 26358 webserver.cc:460] Webserver started at http://127.4.140.254:45863/ using document root <none> and password file <none>
I20260928 02:23:02.767565 26358 fs_manager.cc:359] Metadata directory not provided
I20260928 02:23:02.767715 26358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:23:02.768111 26358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:23:02.771092 26358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "1d6c8914608a4e218b0b9f54c73d75da"
format_stamp: "Formatted at 2026-09-28 02:23:02 on dist-test-slave-h37r"
server_key: "37953d4e3bbdbb31d922e8b508359495"
server_key_iv: "97d024189bb0df23d0fd7d3338e4b033"
server_key_version: "encryptionkey@0"
I20260928 02:23:02.771871 26358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "1d6c8914608a4e218b0b9f54c73d75da"
format_stamp: "Formatted at 2026-09-28 02:23:02 on dist-test-slave-h37r"
server_key: "37953d4e3bbdbb31d922e8b508359495"
server_key_iv: "97d024189bb0df23d0fd7d3338e4b033"
server_key_version: "encryptionkey@0"
I20260928 02:23:02.777247 26358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260928 02:23:02.781355 26373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:23:02.782824 26358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:23:02.783044 26358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "1d6c8914608a4e218b0b9f54c73d75da"
format_stamp: "Formatted at 2026-09-28 02:23:02 on dist-test-slave-h37r"
server_key: "37953d4e3bbdbb31d922e8b508359495"
server_key_iv: "97d024189bb0df23d0fd7d3338e4b033"
server_key_version: "encryptionkey@0"
I20260928 02:23:02.783267 26358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:23:02.799364 26358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:23:02.803619 26358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:23:02.803968 26358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:23:02.823459 26358 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:43405
I20260928 02:23:02.823484 26424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:43405 every 8 connection(s)
I20260928 02:23:02.825018 26358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:23:02.829186 26425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:23:02.834844 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 26358
I20260928 02:23:02.835071 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:23:02.835526 4659 external_mini_cluster.cc:1428] Setting key 1dbf17641197911bf308c29f221fbebf
I20260928 02:23:02.841606 26425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da: Bootstrap starting.
I20260928 02:23:02.845090 26425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da: Neither blocks nor log segments found. Creating new log.
I20260928 02:23:02.846256 26425 log.cc:824] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:23:02 dist-test-slave-h37r krb5kdc[26345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:23:02.849578 26425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da: No bootstrap required, opened a new log
I20260928 02:23:02.858341 26428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:02.837358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50753 (local address 127.4.140.254:43405)
0928 02:23:02.838216 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:02.838229 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:02.838269 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:23:02.840119 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:02.840144 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:02.840564 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:02.840979 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:02.842464 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:02.843797 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:02.845102 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:02.845872 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:02.846084 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:02.850640 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:02.850648 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:02.850663 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:02.850706 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:02.853919 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:02.854759 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:02.854769 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:02.854775 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:02.854960 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:02.855446 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:02.855456 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:02.855458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:02.856074 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:02.856416 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:02.856961 (+ 545us) server_negotiation.cc:299] Negotiation successful
0928 02:23:02.857288 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":395,"threads_started":1}
I20260928 02:23:02.858930 26425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6c8914608a4e218b0b9f54c73d75da" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43405 } }
I20260928 02:23:02.859438 26425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:23:02.859575 26425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d6c8914608a4e218b0b9f54c73d75da, State: Initialized, Role: FOLLOWER
I20260928 02:23:02.860273 26425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6c8914608a4e218b0b9f54c73d75da" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43405 } }
I20260928 02:23:02.860554 26425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:23:02.860730 26425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:23:02.860939 26425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:23:02.862810 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:23:02.863886 26425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6c8914608a4e218b0b9f54c73d75da" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43405 } }
I20260928 02:23:02.864483 26425 leader_election.cc:302] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d6c8914608a4e218b0b9f54c73d75da; no voters:
I20260928 02:23:02.865186 26425 leader_election.cc:288] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:23:02.865324 26430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:23:02.867148 26430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da [term 1 LEADER]: Becoming Leader. State: Replica: 1d6c8914608a4e218b0b9f54c73d75da, State: Running, Role: LEADER
I20260928 02:23:02.867815 26430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6c8914608a4e218b0b9f54c73d75da" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43405 } }
I20260928 02:23:02.868186 26425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:23:02.871627 26434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d6c8914608a4e218b0b9f54c73d75da. Latest consensus state: current_term: 1 leader_uuid: "1d6c8914608a4e218b0b9f54c73d75da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6c8914608a4e218b0b9f54c73d75da" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43405 } } }
I20260928 02:23:02.873538 26434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da [sys.catalog]: This master's current role is: LEADER
I20260928 02:23:02.871170 26433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d6c8914608a4e218b0b9f54c73d75da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6c8914608a4e218b0b9f54c73d75da" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43405 } } }
I20260928 02:23:02.874333 26433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da [sys.catalog]: This master's current role is: LEADER
I20260928 02:23:02.878201 26440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:23:02.890599 26440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:23:02.896888 26428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:02.869090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55029 (local address 127.4.140.254:43405)
0928 02:23:02.869278 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:02.869287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:02.873028 (+ 3741us) server_negotiation.cc:407] Connection header received
0928 02:23:02.873277 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:02.873281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:02.873386 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:02.873524 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:02.875421 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:02.877108 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:02.878988 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:02.880152 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:02.880319 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:02.884831 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:02.884847 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:02.884857 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:02.884926 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:02.892009 (+ 7083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:02.893224 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:02.893230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:02.893233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:02.893405 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:02.894027 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:02.894032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:02.894036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:02.894508 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:02.894810 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:02.896352 (+ 1542us) server_negotiation.cc:299] Negotiation successful
0928 02:23:02.896534 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:23:02.901908 26440 catalog_manager.cc:1347] Generated new cluster ID: 727c44685616492eacb90065ecb06558
I20260928 02:23:02.902041 26440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:23:02.920408 26440 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:23:02.921865 26440 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:23:02.930539 26440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d6c8914608a4e218b0b9f54c73d75da: Generated new TSK 0
I20260928 02:23:02.931661 26440 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:23:02.942121 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 26358
I20260928 02:23:02.943922 26420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:23:03.034044 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 26358
W20260928 02:23:03.051374 26343 connection.cc:535] client connection to 127.4.140.254:43405 recv error: Network error: failed to read from TLS socket (remote: 127.4.140.254:43405): Connection reset by peer (error 104)
2026-09-28T02:23:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2053 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:23:03 dist-test-slave-h37r krb5kdc[26457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:23:03 dist-test-slave-h37r krb5kdc[26457](info): set up 2 sockets
Sep 28 02:23:03 dist-test-slave-h37r krb5kdc[26457](info): commencing operation
krb5kdc: starting...
W20260928 02:23:04.156729 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:23:04 dist-test-slave-h37r krb5kdc[26457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:23:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:23:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:23:04.423442 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:43535
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:40007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:43535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:23:04.759217 26470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:23:04.759719 26470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:23:04.759969 26470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:23:04.768352 26470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:23:04.768478 26470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:23:04.768594 26470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:23:04.768713 26470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:23:04.768800 26470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:23:04.779218 26470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:40007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:43535
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:43535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:23:04.780658 26470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:23:04.781972 26470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:23:04.794420 26475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:04.794502 26478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:04.794461 26476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:23:04.795687 26470 server_base.cc:1034] running on GCE node
I20260928 02:23:04.796553 26470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:23:04.798120 26470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:23:04.799360 26470 hybrid_clock.cc:648] HybridClock initialized: now 1790562184799314 us; error 41 us; skew 500 ppm
I20260928 02:23:04.799724 26470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:23:04 dist-test-slave-h37r krb5kdc[26457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562184, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:23:04.802948 26470 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:23:04.804631 26470 webserver.cc:460] Webserver started at http://127.4.140.254:37181/ using document root <none> and password file <none>
I20260928 02:23:04.805336 26470 fs_manager.cc:359] Metadata directory not provided
I20260928 02:23:04.805493 26470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:23:04.805799 26470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:23:04.809049 26470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "4534809f3b914e21ab358ca000e2f53d"
format_stamp: "Formatted at 2026-09-28 02:23:04 on dist-test-slave-h37r"
server_key: "c3a44b45c8461e5f58a93b065bf3ebe4"
server_key_iv: "341d2c00cd3daad91f4bfa70f6f68a63"
server_key_version: "encryptionkey@0"
I20260928 02:23:04.809906 26470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "4534809f3b914e21ab358ca000e2f53d"
format_stamp: "Formatted at 2026-09-28 02:23:04 on dist-test-slave-h37r"
server_key: "c3a44b45c8461e5f58a93b065bf3ebe4"
server_key_iv: "341d2c00cd3daad91f4bfa70f6f68a63"
server_key_version: "encryptionkey@0"
I20260928 02:23:04.815552 26470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:23:04.819308 26485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:23:04.821106 26470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:23:04.821332 26470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "4534809f3b914e21ab358ca000e2f53d"
format_stamp: "Formatted at 2026-09-28 02:23:04 on dist-test-slave-h37r"
server_key: "c3a44b45c8461e5f58a93b065bf3ebe4"
server_key_iv: "341d2c00cd3daad91f4bfa70f6f68a63"
server_key_version: "encryptionkey@0"
I20260928 02:23:04.821563 26470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:23:04.839958 26470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:23:04.844450 26470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:23:04.844779 26470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:23:04.864025 26470 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:43535
I20260928 02:23:04.864096 26536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:43535 every 8 connection(s)
I20260928 02:23:04.865518 26470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:23:04.869385 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 26470
I20260928 02:23:04.869599 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:23:04.869576 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:23:04.870131 4659 external_mini_cluster.cc:1428] Setting key e98e616fe26c34757283112c71d9c1ce
I20260928 02:23:04.882931 26537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d: Bootstrap starting.
I20260928 02:23:04.883859 26540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:04.872035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33255 (local address 127.4.140.254:43535)
0928 02:23:04.872942 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:04.872955 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:04.873001 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:23:04.874795 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:04.874820 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:04.875273 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:04.875771 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:04.876397 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:04.877700 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:04.879084 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:04.879837 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:04.880141 (+ 304us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:04.881443 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:04.881450 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:04.881467 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:23:04.882066 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:04.882145 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:23:04.882171 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:04.882344 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:04.882616 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 02:23:04.882802 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":368,"threads_started":1}
I20260928 02:23:04.886533 26537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d: Neither blocks nor log segments found. Creating new log.
I20260928 02:23:04.887774 26537 log.cc:824] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:23:04.888067 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:23:04.891377 26537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d: No bootstrap required, opened a new log
I20260928 02:23:04.900867 26540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:04.893323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47175 (local address 127.4.140.254:43535)
0928 02:23:04.893556 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:04.893566 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:04.894056 (+ 490us) server_negotiation.cc:407] Connection header received
0928 02:23:04.894262 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:04.894266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:04.894330 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:04.894423 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:04.895142 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:04.896214 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:04.898202 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:04.898690 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:04.898788 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:04.900071 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:04.900079 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:04.900082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:23:04.900101 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:04.900142 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:23:04.900157 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:04.900239 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:04.900459 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 02:23:04.900508 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:23:04.902220 26537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4534809f3b914e21ab358ca000e2f53d" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43535 } }
I20260928 02:23:04.902633 26537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:23:04.902752 26537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4534809f3b914e21ab358ca000e2f53d, State: Initialized, Role: FOLLOWER
I20260928 02:23:04.903406 26537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4534809f3b914e21ab358ca000e2f53d" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43535 } }
I20260928 02:23:04.903656 26537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:23:04.903810 26537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:23:04.904060 26537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:23:04.906288 26537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4534809f3b914e21ab358ca000e2f53d" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43535 } }
I20260928 02:23:04.906857 26537 leader_election.cc:302] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4534809f3b914e21ab358ca000e2f53d; no voters:
I20260928 02:23:04.907446 26537 leader_election.cc:288] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:23:04.907590 26547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:23:04.908367 26547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d [term 1 LEADER]: Becoming Leader. State: Replica: 4534809f3b914e21ab358ca000e2f53d, State: Running, Role: LEADER
I20260928 02:23:04.908890 26547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4534809f3b914e21ab358ca000e2f53d" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43535 } }
I20260928 02:23:04.909677 26537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:23:04.911775 26548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4534809f3b914e21ab358ca000e2f53d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4534809f3b914e21ab358ca000e2f53d" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43535 } } }
I20260928 02:23:04.912276 26548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d [sys.catalog]: This master's current role is: LEADER
I20260928 02:23:04.911775 26549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4534809f3b914e21ab358ca000e2f53d. Latest consensus state: current_term: 1 leader_uuid: "4534809f3b914e21ab358ca000e2f53d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4534809f3b914e21ab358ca000e2f53d" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 43535 } } }
I20260928 02:23:04.913681 26549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d [sys.catalog]: This master's current role is: LEADER
I20260928 02:23:04.921093 26552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:23:04.926972 26552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:23:04.937176 26552 catalog_manager.cc:1347] Generated new cluster ID: 2cd79000ec47431a8d937ca0b1a1a3db
I20260928 02:23:04.937325 26552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:23:04.962620 26552 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:23:04.964120 26552 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:23:04.973806 26552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4534809f3b914e21ab358ca000e2f53d: Generated new TSK 0
I20260928 02:23:04.975039 26552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:23:04.993520 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 26470
W20260928 02:23:04.995311 26532 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33255
W20260928 02:23:04.996160 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 26470: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:23:04.996317 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 26470
W20260928 02:23:05.010833 26455 connection.cc:535] client connection to 127.4.140.254:43535 recv error: Network error: failed to read from TLS socket (remote: 127.4.140.254:43535): Connection reset by peer (error 104)
2026-09-28T02:23:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1958 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:23:05 dist-test-slave-h37r krb5kdc[26568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:23:05 dist-test-slave-h37r krb5kdc[26568](info): set up 2 sockets
Sep 28 02:23:05 dist-test-slave-h37r krb5kdc[26568](info): commencing operation
krb5kdc: starting...
W20260928 02:23:06.104789 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:23:06 dist-test-slave-h37r krb5kdc[26568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:23:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:23:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:23:06.379767 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:38397
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:40737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:38397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:23:06.716984 26581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:23:06.717279 26581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:23:06.717468 26581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:23:06.724745 26581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:23:06.724890 26581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:23:06.725001 26581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:23:06.725114 26581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:23:06.725205 26581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:23:06.735257 26581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:40737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:38397
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:38397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:23:06.736640 26581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:23:06.738010 26581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:23:06.749997 26586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:06.750042 26587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:06.750063 26589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:23:06.751328 26581 server_base.cc:1034] running on GCE node
I20260928 02:23:06.752234 26581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:23:06.753774 26581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:23:06.755062 26581 hybrid_clock.cc:648] HybridClock initialized: now 1790562186754971 us; error 54 us; skew 500 ppm
I20260928 02:23:06.755576 26581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:23:06 dist-test-slave-h37r krb5kdc[26568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562186, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:23:06.759351 26581 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:23:06.761425 26581 webserver.cc:460] Webserver started at http://127.4.140.254:44713/ using document root <none> and password file <none>
I20260928 02:23:06.762389 26581 fs_manager.cc:359] Metadata directory not provided
I20260928 02:23:06.762557 26581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:23:06.762951 26581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:23:06.766681 26581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "5114d2c57129458189128426f94ca286"
format_stamp: "Formatted at 2026-09-28 02:23:06 on dist-test-slave-h37r"
server_key: "d93339aecd11bbe7d94f64fa457f3e16"
server_key_iv: "3713dc389e5bd16ede6e9db909741ca3"
server_key_version: "encryptionkey@0"
I20260928 02:23:06.767498 26581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "5114d2c57129458189128426f94ca286"
format_stamp: "Formatted at 2026-09-28 02:23:06 on dist-test-slave-h37r"
server_key: "d93339aecd11bbe7d94f64fa457f3e16"
server_key_iv: "3713dc389e5bd16ede6e9db909741ca3"
server_key_version: "encryptionkey@0"
I20260928 02:23:06.772891 26581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 02:23:06.776506 26596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:23:06.777871 26581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:23:06.778071 26581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "5114d2c57129458189128426f94ca286"
format_stamp: "Formatted at 2026-09-28 02:23:06 on dist-test-slave-h37r"
server_key: "d93339aecd11bbe7d94f64fa457f3e16"
server_key_iv: "3713dc389e5bd16ede6e9db909741ca3"
server_key_version: "encryptionkey@0"
I20260928 02:23:06.778276 26581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:23:06.784251 26581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:23:06.796077 26581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:23:06.796401 26581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:23:06.816035 26581 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:38397
I20260928 02:23:06.816093 26647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:38397 every 8 connection(s)
I20260928 02:23:06.817509 26581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:23:06.821382 26648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:23:06.825502 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 26581
I20260928 02:23:06.825709 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:23:06.826141 4659 external_mini_cluster.cc:1428] Setting key f3191384e73b91cdf3654ed06f55143c
I20260928 02:23:06.832890 26648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286: Bootstrap starting.
I20260928 02:23:06.834718 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:06.828075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50361 (local address 127.4.140.254:38397)
0928 02:23:06.828907 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:06.828920 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:06.828966 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:23:06.831204 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:06.831227 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:06.831591 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:06.831995 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:06.833082 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:06.833089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:06.833101 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:23:06.833116 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:06.833176 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:23:06.833199 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:06.833388 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:06.833547 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 02:23:06.833758 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":360,"threads_started":1}
I20260928 02:23:06.836404 26648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286: Neither blocks nor log segments found. Creating new log.
I20260928 02:23:06.837613 26648 log.cc:824] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286: Log is configured to *not* fsync() on all Append() calls
I20260928 02:23:06.838543 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:23:06.840516 26648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286: No bootstrap required, opened a new log
I20260928 02:23:06.845628 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:06.842972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38551 (local address 127.4.140.254:38397)
0928 02:23:06.843134 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:06.843138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:06.843583 (+ 445us) server_negotiation.cc:407] Connection header received
0928 02:23:06.843781 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:06.843793 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:06.843842 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:06.843998 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:06.844947 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:06.844953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:06.844958 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:23:06.844977 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:06.845030 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:23:06.845049 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:06.845179 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:06.845277 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 02:23:06.845342 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:23:06.849022 26648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5114d2c57129458189128426f94ca286" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 38397 } }
I20260928 02:23:06.849418 26648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:23:06.849539 26648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5114d2c57129458189128426f94ca286, State: Initialized, Role: FOLLOWER
I20260928 02:23:06.850184 26648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5114d2c57129458189128426f94ca286" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 38397 } }
I20260928 02:23:06.850436 26648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:23:06.850617 26648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:23:06.850800 26648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:23:06.853441 26648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5114d2c57129458189128426f94ca286" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 38397 } }
I20260928 02:23:06.853979 26648 leader_election.cc:302] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5114d2c57129458189128426f94ca286; no voters:
I20260928 02:23:06.854597 26648 leader_election.cc:288] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:23:06.854682 26658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:23:06.855494 26658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286 [term 1 LEADER]: Becoming Leader. State: Replica: 5114d2c57129458189128426f94ca286, State: Running, Role: LEADER
I20260928 02:23:06.856206 26658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5114d2c57129458189128426f94ca286" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 38397 } }
I20260928 02:23:06.857025 26648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:23:06.859324 26659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5114d2c57129458189128426f94ca286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5114d2c57129458189128426f94ca286" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 38397 } } }
I20260928 02:23:06.859669 26659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286 [sys.catalog]: This master's current role is: LEADER
I20260928 02:23:06.860041 26660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5114d2c57129458189128426f94ca286. Latest consensus state: current_term: 1 leader_uuid: "5114d2c57129458189128426f94ca286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5114d2c57129458189128426f94ca286" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 38397 } } }
I20260928 02:23:06.860423 26660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286 [sys.catalog]: This master's current role is: LEADER
W20260928 02:23:06.871035 26674 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:23:06.871205 26674 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:23:06.871348 26666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:23:06.873113 26666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:23:06.881986 26666 catalog_manager.cc:1347] Generated new cluster ID: da2cf05984a042f599fa68280565b22b
I20260928 02:23:06.882102 26666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:23:06.909384 26666 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:23:06.909881 26666 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:23:06.921461 26666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5114d2c57129458189128426f94ca286: Generated new TSK 0
I20260928 02:23:06.922410 26666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:23:06.928978 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 26581
W20260928 02:23:06.930115 26643 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50361
W20260928 02:23:06.930796 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 26581: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:23:06.930939 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 26581
2026-09-28T02:23:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1937 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:23:07 dist-test-slave-h37r krb5kdc[26679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:23:07 dist-test-slave-h37r krb5kdc[26679](info): set up 2 sockets
Sep 28 02:23:07 dist-test-slave-h37r krb5kdc[26679](info): commencing operation
krb5kdc: starting...
W20260928 02:23:08.048744 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:23:08 dist-test-slave-h37r krb5kdc[26679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:23:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:23:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:23:08.313611 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:46721
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:46721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:23:08.648481 26692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:23:08.648896 26692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:23:08.649075 26692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:23:08.656481 26692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:23:08.656664 26692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:23:08.656787 26692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:23:08.656896 26692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:23:08.656986 26692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:23:08.667254 26692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:46721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:46721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:23:08.668704 26692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:23:08.670058 26692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:23:08.682801 26698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:08.682868 26700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:08.682878 26697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:23:08.683782 26692 server_base.cc:1034] running on GCE node
I20260928 02:23:08.684629 26692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:23:08.686125 26692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:23:08.687377 26692 hybrid_clock.cc:648] HybridClock initialized: now 1790562188687328 us; error 43 us; skew 500 ppm
I20260928 02:23:08.687745 26692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:23:08 dist-test-slave-h37r krb5kdc[26679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562188, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:23:08.690891 26692 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:23:08.692539 26692 webserver.cc:460] Webserver started at http://127.4.140.254:33307/ using document root <none> and password file <none>
I20260928 02:23:08.693220 26692 fs_manager.cc:359] Metadata directory not provided
I20260928 02:23:08.693377 26692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:23:08.693672 26692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:23:08.696846 26692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "7790d8d6703a442d823d132657502e95"
format_stamp: "Formatted at 2026-09-28 02:23:08 on dist-test-slave-h37r"
server_key: "ea7eff61597e4ce0b3d420a36fee21db"
server_key_iv: "640b8459bc1b9a752cfed2dd3d78f034"
server_key_version: "encryptionkey@0"
I20260928 02:23:08.697628 26692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "7790d8d6703a442d823d132657502e95"
format_stamp: "Formatted at 2026-09-28 02:23:08 on dist-test-slave-h37r"
server_key: "ea7eff61597e4ce0b3d420a36fee21db"
server_key_iv: "640b8459bc1b9a752cfed2dd3d78f034"
server_key_version: "encryptionkey@0"
I20260928 02:23:08.702719 26692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:23:08.706147 26707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:23:08.707442 26692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:23:08.707628 26692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "7790d8d6703a442d823d132657502e95"
format_stamp: "Formatted at 2026-09-28 02:23:08 on dist-test-slave-h37r"
server_key: "ea7eff61597e4ce0b3d420a36fee21db"
server_key_iv: "640b8459bc1b9a752cfed2dd3d78f034"
server_key_version: "encryptionkey@0"
I20260928 02:23:08.707823 26692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:23:08.727005 26692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:23:08.731097 26692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:23:08.731403 26692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:23:08.750089 26692 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:46721
I20260928 02:23:08.750108 26758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:46721 every 8 connection(s)
I20260928 02:23:08.751472 26692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:23:08.755651 26759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:23:08.759400 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 26692
I20260928 02:23:08.759577 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:23:08.760133 4659 external_mini_cluster.cc:1428] Setting key c054d54b735466ca99fe0a8945c40bf1
I20260928 02:23:08.769584 26759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95: Bootstrap starting.
I20260928 02:23:08.773361 26759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95: Neither blocks nor log segments found. Creating new log.
I20260928 02:23:08.774703 26759 log.cc:824] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:23:08 dist-test-slave-h37r krb5kdc[26679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:23:08.777820 26759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95: No bootstrap required, opened a new log
I20260928 02:23:08.783483 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:08.762306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48189 (local address 127.4.140.254:46721)
0928 02:23:08.763151 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:08.763162 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:08.763203 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:23:08.764945 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:08.764968 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:08.765366 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:08.765690 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:08.767471 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:08.768850 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:08.770649 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:08.771346 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:08.771472 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:08.776437 (+ 4965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:08.776445 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:08.776458 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:08.776490 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:08.779546 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:08.780400 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:08.780407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:08.780413 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:08.780579 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:08.780962 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:08.780965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:08.780967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:08.781452 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:08.781688 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:08.782091 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 02:23:08.782450 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":401,"threads_started":1}
I20260928 02:23:08.786613 26759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7790d8d6703a442d823d132657502e95" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 46721 } }
I20260928 02:23:08.786971 26759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:23:08.787091 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:23:08.787098 26759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7790d8d6703a442d823d132657502e95, State: Initialized, Role: FOLLOWER
I20260928 02:23:08.787622 26759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7790d8d6703a442d823d132657502e95" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 46721 } }
I20260928 02:23:08.787864 26759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:23:08.788048 26759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:23:08.788210 26759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:23:08.790558 26759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7790d8d6703a442d823d132657502e95" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 46721 } }
I20260928 02:23:08.791056 26759 leader_election.cc:302] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7790d8d6703a442d823d132657502e95; no voters:
I20260928 02:23:08.791708 26759 leader_election.cc:288] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:23:08.791769 26768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:23:08.793373 26768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95 [term 1 LEADER]: Becoming Leader. State: Replica: 7790d8d6703a442d823d132657502e95, State: Running, Role: LEADER
I20260928 02:23:08.793957 26768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7790d8d6703a442d823d132657502e95" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 46721 } }
I20260928 02:23:08.796541 26770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7790d8d6703a442d823d132657502e95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7790d8d6703a442d823d132657502e95" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 46721 } } }
I20260928 02:23:08.796934 26770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95 [sys.catalog]: This master's current role is: LEADER
I20260928 02:23:08.798439 26769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7790d8d6703a442d823d132657502e95. Latest consensus state: current_term: 1 leader_uuid: "7790d8d6703a442d823d132657502e95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7790d8d6703a442d823d132657502e95" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 46721 } } }
I20260928 02:23:08.798732 26769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95 [sys.catalog]: This master's current role is: LEADER
I20260928 02:23:08.801167 26759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:23:08.802035 26773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:23:08.808072 26773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:23:08.811954 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:08.791774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33233 (local address 127.4.140.254:46721)
0928 02:23:08.792132 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:08.792140 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:08.792912 (+ 772us) server_negotiation.cc:407] Connection header received
0928 02:23:08.793147 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:08.793156 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:08.793266 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:08.793415 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:08.796081 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:08.797661 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:08.799629 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:08.800300 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:08.800421 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:08.803849 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:08.803859 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:08.803867 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:08.803941 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:08.807728 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:08.808881 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:08.808887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:08.808894 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:08.809040 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:08.809519 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:08.809523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:08.809526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:08.810005 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:08.811221 (+ 1216us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:08.811377 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 02:23:08.811595 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260928 02:23:08.818183 26773 catalog_manager.cc:1347] Generated new cluster ID: 0ec509c90b744aeb8c5d77c354432808
I20260928 02:23:08.818317 26773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:23:08.830715 26773 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:23:08.832254 26773 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:23:08.846707 26773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7790d8d6703a442d823d132657502e95: Generated new TSK 0
I20260928 02:23:08.847613 26773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:23:08.855957 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 26692
I20260928 02:23:08.857121 26754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:23:08.952260 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 26692
2026-09-28T02:23:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2021 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:23:09 dist-test-slave-h37r krb5kdc[26792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:23:09 dist-test-slave-h37r krb5kdc[26792](info): set up 2 sockets
Sep 28 02:23:09 dist-test-slave-h37r krb5kdc[26792](info): commencing operation
krb5kdc: starting...
W20260928 02:23:10.068467 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:23:10 dist-test-slave-h37r krb5kdc[26792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:23:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:23:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:23:10.339432 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:42429
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:42429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:23:10.676128 26805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:23:10.676481 26805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:23:10.676643 26805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:23:10.684126 26805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:23:10.684281 26805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:23:10.684383 26805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:23:10.684492 26805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:23:10.684568 26805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:23:10.694832 26805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:42429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:42429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:23:10.696281 26805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:23:10.697623 26805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:23:10.711861 26810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:10.711984 26813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:10.711956 26811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:23:10.713248 26805 server_base.cc:1034] running on GCE node
I20260928 02:23:10.714083 26805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:23:10.715636 26805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:23:10.716848 26805 hybrid_clock.cc:648] HybridClock initialized: now 1790562190716782 us; error 56 us; skew 500 ppm
I20260928 02:23:10.717226 26805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:23:10 dist-test-slave-h37r krb5kdc[26792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562190, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:23:10.720579 26805 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:23:10.722062 26805 webserver.cc:460] Webserver started at http://127.4.140.254:40163/ using document root <none> and password file <none>
I20260928 02:23:10.722707 26805 fs_manager.cc:359] Metadata directory not provided
I20260928 02:23:10.722805 26805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:23:10.723066 26805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:23:10.726577 26805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "c33b030192804b17a2db50bbb9e0dd9a"
format_stamp: "Formatted at 2026-09-28 02:23:10 on dist-test-slave-h37r"
server_key: "e9457d0b95a98542bdf253d279bff8dd"
server_key_iv: "fa9e34f0875cc8fa8788609f97a8b0ac"
server_key_version: "encryptionkey@0"
I20260928 02:23:10.727332 26805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "c33b030192804b17a2db50bbb9e0dd9a"
format_stamp: "Formatted at 2026-09-28 02:23:10 on dist-test-slave-h37r"
server_key: "e9457d0b95a98542bdf253d279bff8dd"
server_key_iv: "fa9e34f0875cc8fa8788609f97a8b0ac"
server_key_version: "encryptionkey@0"
I20260928 02:23:10.732689 26805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 02:23:10.736182 26820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:23:10.737551 26805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:23:10.737756 26805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "c33b030192804b17a2db50bbb9e0dd9a"
format_stamp: "Formatted at 2026-09-28 02:23:10 on dist-test-slave-h37r"
server_key: "e9457d0b95a98542bdf253d279bff8dd"
server_key_iv: "fa9e34f0875cc8fa8788609f97a8b0ac"
server_key_version: "encryptionkey@0"
I20260928 02:23:10.737964 26805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:23:10.759696 26805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:23:10.769196 26805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:23:10.769462 26805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:23:10.787874 26805 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:42429
I20260928 02:23:10.787881 26871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:42429 every 8 connection(s)
I20260928 02:23:10.789527 26805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:23:10.793439 26872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:23:10.796281 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 26805
I20260928 02:23:10.796470 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:23:10.796911 4659 external_mini_cluster.cc:1428] Setting key c36f5721bf83af6897d879f85395d2f7
I20260928 02:23:10.807351 26872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a: Bootstrap starting.
I20260928 02:23:10.811748 26872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a: Neither blocks nor log segments found. Creating new log.
I20260928 02:23:10.813247 26872 log.cc:824] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:23:10.813652 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:10.798863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51549 (local address 127.4.140.254:42429)
0928 02:23:10.799833 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:10.799850 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:10.799920 (+ 70us) server_negotiation.cc:407] Connection header received
0928 02:23:10.802334 (+ 2414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:10.802364 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:10.802936 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:10.803424 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:10.804014 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:10.806154 (+ 2140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:10.807853 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:10.808915 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:10.809143 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:10.810492 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:10.810502 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:10.810521 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:23:10.811288 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:10.811399 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:23:10.811437 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:10.811559 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:10.811844 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 02:23:10.812198 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":460,"threads_started":1}
I20260928 02:23:10.817811 26872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a: No bootstrap required, opened a new log
I20260928 02:23:10.818017 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:23:10.829146 26872 raft_consensus.cc:348] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33b030192804b17a2db50bbb9e0dd9a" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42429 } }
I20260928 02:23:10.829738 26872 raft_consensus.cc:374] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:23:10.829895 26872 raft_consensus.cc:729] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33b030192804b17a2db50bbb9e0dd9a, State: Initialized, Role: FOLLOWER
I20260928 02:23:10.830684 26872 consensus_queue.cc:260] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33b030192804b17a2db50bbb9e0dd9a" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42429 } }
I20260928 02:23:10.830843 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:10.823423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59513 (local address 127.4.140.254:42429)
0928 02:23:10.823739 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:10.823746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:10.824154 (+ 408us) server_negotiation.cc:407] Connection header received
0928 02:23:10.824355 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:10.824359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:10.824426 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:10.824525 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:10.825238 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:10.826017 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:10.828031 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:10.828485 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:10.828582 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:10.829937 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:10.829945 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:10.829953 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:23:10.829974 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:10.830039 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:23:10.830059 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:10.830151 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:10.830365 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 02:23:10.830475 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 02:23:10.830997 26872 raft_consensus.cc:388] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:23:10.831164 26872 raft_consensus.cc:482] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:23:10.831372 26872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:23:10.833972 26872 raft_consensus.cc:504] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33b030192804b17a2db50bbb9e0dd9a" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42429 } }
I20260928 02:23:10.834542 26872 leader_election.cc:302] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c33b030192804b17a2db50bbb9e0dd9a; no voters:
I20260928 02:23:10.835254 26872 leader_election.cc:288] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:23:10.835382 26882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:23:10.836493 26882 raft_consensus.cc:686] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a [term 1 LEADER]: Becoming Leader. State: Replica: c33b030192804b17a2db50bbb9e0dd9a, State: Running, Role: LEADER
I20260928 02:23:10.837114 26882 consensus_queue.cc:237] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33b030192804b17a2db50bbb9e0dd9a" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42429 } }
I20260928 02:23:10.837920 26872 sys_catalog.cc:564] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:23:10.841487 26884 sys_catalog.cc:455] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c33b030192804b17a2db50bbb9e0dd9a. Latest consensus state: current_term: 1 leader_uuid: "c33b030192804b17a2db50bbb9e0dd9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33b030192804b17a2db50bbb9e0dd9a" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42429 } } }
I20260928 02:23:10.841792 26884 sys_catalog.cc:458] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a [sys.catalog]: This master's current role is: LEADER
I20260928 02:23:10.841128 26883 sys_catalog.cc:455] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c33b030192804b17a2db50bbb9e0dd9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33b030192804b17a2db50bbb9e0dd9a" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42429 } } }
I20260928 02:23:10.842437 26883 sys_catalog.cc:458] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a [sys.catalog]: This master's current role is: LEADER
I20260928 02:23:10.843186 26891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:23:10.848039 26891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:23:10.858026 26891 catalog_manager.cc:1347] Generated new cluster ID: eadadcf394514e798c264f837d972d05
I20260928 02:23:10.858173 26891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:23:10.887549 26891 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:23:10.889639 26891 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:23:10.907133 26891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c33b030192804b17a2db50bbb9e0dd9a: Generated new TSK 0
I20260928 02:23:10.908241 26891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:23:10.924044 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 26805
W20260928 02:23:10.925318 26867 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51549
W20260928 02:23:10.926057 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 26805: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:23:10.926213 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 26805
2026-09-28T02:23:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1973 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:23:11 dist-test-slave-h37r krb5kdc[26903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:23:11 dist-test-slave-h37r krb5kdc[26903](info): set up 2 sockets
Sep 28 02:23:11 dist-test-slave-h37r krb5kdc[26903](info): commencing operation
krb5kdc: starting...
W20260928 02:23:12.044523 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:23:12 dist-test-slave-h37r krb5kdc[26903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:23:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:23:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:23:12.315408 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:37025
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:37025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:23:12.658497 26916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:23:12.658790 26916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:23:12.658960 26916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:23:12.666358 26916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:23:12.666473 26916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:23:12.666574 26916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:23:12.666659 26916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:23:12.666747 26916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:23:12.680684 26916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:37025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:37025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:23:12.682060 26916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:23:12.683394 26916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:23:12.695446 26921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:12.695489 26922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:12.695662 26924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:23:12.696447 26916 server_base.cc:1034] running on GCE node
I20260928 02:23:12.697162 26916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:23:12.698587 26916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:23:12.699833 26916 hybrid_clock.cc:648] HybridClock initialized: now 1790562192699796 us; error 30 us; skew 500 ppm
I20260928 02:23:12.700198 26916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:23:12 dist-test-slave-h37r krb5kdc[26903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562192, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:23:12.708593 26916 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:23:12.710170 26916 webserver.cc:460] Webserver started at http://127.4.140.254:39051/ using document root <none> and password file <none>
I20260928 02:23:12.710881 26916 fs_manager.cc:359] Metadata directory not provided
I20260928 02:23:12.711035 26916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:23:12.711352 26916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:23:12.714429 26916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "c8623dd3393c4155bef2edcb61a89ed0"
format_stamp: "Formatted at 2026-09-28 02:23:12 on dist-test-slave-h37r"
server_key: "3f07d52efe1a37484f6b7f596deb0191"
server_key_iv: "58aac322e1319c38350278acc5d176c5"
server_key_version: "encryptionkey@0"
I20260928 02:23:12.715240 26916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "c8623dd3393c4155bef2edcb61a89ed0"
format_stamp: "Formatted at 2026-09-28 02:23:12 on dist-test-slave-h37r"
server_key: "3f07d52efe1a37484f6b7f596deb0191"
server_key_iv: "58aac322e1319c38350278acc5d176c5"
server_key_version: "encryptionkey@0"
I20260928 02:23:12.720724 26916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 02:23:12.724269 26931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:23:12.725605 26916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:23:12.725795 26916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "c8623dd3393c4155bef2edcb61a89ed0"
format_stamp: "Formatted at 2026-09-28 02:23:12 on dist-test-slave-h37r"
server_key: "3f07d52efe1a37484f6b7f596deb0191"
server_key_iv: "58aac322e1319c38350278acc5d176c5"
server_key_version: "encryptionkey@0"
I20260928 02:23:12.725992 26916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:23:12.731851 26916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:23:12.737145 26916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:23:12.737483 26916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:23:12.760957 26916 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:37025
I20260928 02:23:12.760964 26982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:37025 every 8 connection(s)
I20260928 02:23:12.762864 26916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:23:12.767841 26983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:23:12.772636 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 26916
I20260928 02:23:12.772823 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:23:12.773348 4659 external_mini_cluster.cc:1428] Setting key 152dff04d4301d626541557347c12bbb
I20260928 02:23:12.783430 26983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0: Bootstrap starting.
I20260928 02:23:12.783577 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:12.775291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40361 (local address 127.4.140.254:37025)
0928 02:23:12.776157 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:12.776174 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:12.776224 (+ 50us) server_negotiation.cc:407] Connection header received
0928 02:23:12.779047 (+ 2823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:12.779085 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:12.779618 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:12.780176 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:12.781334 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:12.781345 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:12.781370 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:23:12.781393 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:12.781475 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:23:12.781512 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:12.781772 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:12.781914 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 02:23:12.782136 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":392,"threads_started":1}
I20260928 02:23:12.786749 26983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0: Neither blocks nor log segments found. Creating new log.
I20260928 02:23:12.787793 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:23:12.788136 26983 log.cc:824] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:23:12.791038 26983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0: No bootstrap required, opened a new log
I20260928 02:23:12.795030 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:12.792231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49767 (local address 127.4.140.254:37025)
0928 02:23:12.792423 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:12.792431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:12.792944 (+ 513us) server_negotiation.cc:407] Connection header received
0928 02:23:12.793100 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:12.793103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:12.793150 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:12.793234 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:12.794208 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:12.794214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:12.794218 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:23:12.794235 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:12.794280 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:23:12.794295 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:12.794386 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:12.794586 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 02:23:12.794704 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:23:12.799858 26983 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8623dd3393c4155bef2edcb61a89ed0" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 37025 } }
I20260928 02:23:12.800239 26983 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:23:12.800379 26983 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8623dd3393c4155bef2edcb61a89ed0, State: Initialized, Role: FOLLOWER
I20260928 02:23:12.800948 26983 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8623dd3393c4155bef2edcb61a89ed0" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 37025 } }
I20260928 02:23:12.801182 26983 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:23:12.801332 26983 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:23:12.801494 26983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:23:12.803592 26983 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8623dd3393c4155bef2edcb61a89ed0" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 37025 } }
I20260928 02:23:12.804143 26983 leader_election.cc:302] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8623dd3393c4155bef2edcb61a89ed0; no voters:
I20260928 02:23:12.804719 26983 leader_election.cc:288] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:23:12.804876 26993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:23:12.805953 26993 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0 [term 1 LEADER]: Becoming Leader. State: Replica: c8623dd3393c4155bef2edcb61a89ed0, State: Running, Role: LEADER
I20260928 02:23:12.806492 26993 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8623dd3393c4155bef2edcb61a89ed0" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 37025 } }
I20260928 02:23:12.807120 26983 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:23:12.809311 26994 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8623dd3393c4155bef2edcb61a89ed0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8623dd3393c4155bef2edcb61a89ed0" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 37025 } } }
I20260928 02:23:12.809741 26994 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:23:12.809417 26995 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8623dd3393c4155bef2edcb61a89ed0. Latest consensus state: current_term: 1 leader_uuid: "c8623dd3393c4155bef2edcb61a89ed0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8623dd3393c4155bef2edcb61a89ed0" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 37025 } } }
I20260928 02:23:12.810045 26995 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:23:12.812647 26999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:23:12.817834 26999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:23:12.828662 26999 catalog_manager.cc:1347] Generated new cluster ID: e697f686c26c41dab83e4bafb1f78f81
I20260928 02:23:12.828795 26999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:23:12.846259 26999 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:23:12.846704 26999 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:23:12.855556 26999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8623dd3393c4155bef2edcb61a89ed0: Generated new TSK 0
I20260928 02:23:12.856549 26999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:23:12.862202 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 26916
W20260928 02:23:12.863590 26978 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40361
W20260928 02:23:12.864360 4659 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 26916: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:23:12.864519 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 26916
2026-09-28T02:23:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1938 ms)
[----------] 18 tests from AuthTokenIssuingTest (35780 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:23:12 dist-test-slave-h37r krb5kdc[27014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:23:12 dist-test-slave-h37r krb5kdc[27014](info): set up 2 sockets
Sep 28 02:23:12 dist-test-slave-h37r krb5kdc[27014](info): commencing operation
krb5kdc: starting...
W20260928 02:23:13.976743 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:23:14 dist-test-slave-h37r krb5kdc[27014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:23:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:23:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:23:14.249864 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:45357
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:45357,127.4.140.253:43893,127.4.140.252:38129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:23:14.587003 27027 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:23:14.587407 27027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:23:14.587532 27027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:23:14.587666 27027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:23:14.595238 27027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:23:14.595381 27027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:23:14.595477 27027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:23:14.595572 27027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:23:14.595660 27027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:23:14.606027 27027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:45357,127.4.140.253:43893,127.4.140.252:38129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:45357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:23:14.607388 27027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:23:14.608749 27027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:23:14.620925 27032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:14.621057 27035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:14.620896 27033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:23:14.621733 27027 server_base.cc:1034] running on GCE node
I20260928 02:23:14.622534 27027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:23:14.624015 27027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:23:14.625252 27027 hybrid_clock.cc:648] HybridClock initialized: now 1790562194625189 us; error 42 us; skew 500 ppm
I20260928 02:23:14.625608 27027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:23:14 dist-test-slave-h37r krb5kdc[27014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562194, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:23:14.628871 27027 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:23:14.630288 27027 webserver.cc:460] Webserver started at http://127.4.140.254:35799/ using document root <none> and password file <none>
I20260928 02:23:14.630970 27027 fs_manager.cc:359] Metadata directory not provided
I20260928 02:23:14.631054 27027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:23:14.631314 27027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:23:14.634305 27027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "fb9ab4df0efd456f906090a08889d9ee"
format_stamp: "Formatted at 2026-09-28 02:23:14 on dist-test-slave-h37r"
server_key: "f682fbe8926ee2933ae957a6e637c2ef"
server_key_iv: "c240c3d2d22e6c208f3c6ab393413152"
server_key_version: "encryptionkey@0"
I20260928 02:23:14.635059 27027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "fb9ab4df0efd456f906090a08889d9ee"
format_stamp: "Formatted at 2026-09-28 02:23:14 on dist-test-slave-h37r"
server_key: "f682fbe8926ee2933ae957a6e637c2ef"
server_key_iv: "c240c3d2d22e6c208f3c6ab393413152"
server_key_version: "encryptionkey@0"
I20260928 02:23:14.640213 27027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:23:14.643633 27042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:23:14.645190 27027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:23:14.645386 27027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "fb9ab4df0efd456f906090a08889d9ee"
format_stamp: "Formatted at 2026-09-28 02:23:14 on dist-test-slave-h37r"
server_key: "f682fbe8926ee2933ae957a6e637c2ef"
server_key_iv: "c240c3d2d22e6c208f3c6ab393413152"
server_key_version: "encryptionkey@0"
I20260928 02:23:14.645623 27027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:23:14.671999 27027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:23:14.676309 27027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:23:14.676604 27027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:23:14.695065 27027 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:45357
I20260928 02:23:14.695101 27093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:45357 every 8 connection(s)
I20260928 02:23:14.696534 27027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:23:14.700398 27094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:23:14.706598 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 27027
I20260928 02:23:14.706820 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:23:14.707450 4659 external_mini_cluster.cc:1428] Setting key dca8d1c2b844c8b910c37d8ccc1de8c5
I20260928 02:23:14.706990 27094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:23:14 dist-test-slave-h37r krb5kdc[27014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562194, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:23:14.731716 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:14.710185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:45357 (local address 127.0.0.1:53806)
0928 02:23:14.711005 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:14.711059 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:23:14.713141 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:23:14.714150 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:23:14.714156 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:23:14.714728 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:23:14.715104 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:14.715123 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:14.717184 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:14.717192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:14.719161 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:14.719185 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:14.720190 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:14.720200 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:14.720453 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:14.721191 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:23:14.721229 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:23:14.724001 (+ 2772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:23:14.727315 (+ 3314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:14.727328 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:14.727341 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:14.727797 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:14.728244 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:14.728249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:14.728253 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:14.728411 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:14.729174 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:23:14.729184 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:23:14.729407 (+ 223us) client_negotiation.cc:769] Sending connection context
0928 02:23:14.729663 (+ 256us) client_negotiation.cc:240] Negotiation successful
0928 02:23:14.730344 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":210,"thread_start_us":336,"threads_started":1}
I20260928 02:23:14.731765 27097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:14.711490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.4.140.254:45357)
0928 02:23:14.712215 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:14.712224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:14.712281 (+ 57us) server_negotiation.cc:407] Connection header received
0928 02:23:14.713567 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:14.713577 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:14.713901 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:14.714139 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:14.715414 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:14.717008 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:14.719321 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:14.719882 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:14.720088 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:14.724233 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:14.724241 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:14.724254 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:14.724287 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:14.727152 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:14.727948 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:14.727957 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:14.727960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:14.728121 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:14.728531 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:14.728536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:14.728539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:14.729032 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:14.729243 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:14.729803 (+ 560us) server_negotiation.cc:299] Negotiation successful
0928 02:23:14.730222 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":500,"thread_start_us":313,"threads_started":1}
WARNING: no policy specified for kudu/127.4.140.253@KRBTEST.COM; defaulting to no policy
I20260928 02:23:14.734822 27094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.253@KRBTEST.COM" created.
I20260928 02:23:14.736117 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:14.735493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:43893 (local address 127.0.0.1:60570)
0928 02:23:14.735770 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:14.735938 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260928 02:23:14.736352 27045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.140.253:43893: connect: Connection refused (error 111)
I20260928 02:23:14.737876 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:14.737518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:43893 (local address 127.0.0.1:60584)
0928 02:23:14.737677 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:14.737741 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 02:23:14.738058 27094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.253:43893: Network error: Client connection negotiation failed: client connection to 127.4.140.253:43893: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.253.keytab.
Entry for principal kudu/127.4.140.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.253.keytab.
I20260928 02:23:14.771025 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.253:43893
--webserver_interface=127.4.140.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.253.keytab
--principal=kudu/127.4.140.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:45357,127.4.140.253:43893,127.4.140.252:38129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:23:14.787410 27094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } attempt: 1
I20260928 02:23:14.788807 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:14.788182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:43893 (local address 127.0.0.1:60596)
0928 02:23:14.788457 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:14.788616 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 02:23:14.790095 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:14.789600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:43893 (local address 127.0.0.1:60600)
0928 02:23:14.789792 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:14.789897 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 02:23:14.790360 27094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.253:43893: Network error: Client connection negotiation failed: client connection to 127.4.140.253:43893: connect: Connection refused (error 111)
I20260928 02:23:14.858665 27094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } attempt: 2
I20260928 02:23:14.860051 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:14.859400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:43893 (local address 127.0.0.1:60606)
0928 02:23:14.859691 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:14.859824 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 02:23:14.861304 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:14.860875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:43893 (local address 127.0.0.1:60610)
0928 02:23:14.861052 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:14.861142 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 02:23:14.861519 27094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.253:43893: Network error: Client connection negotiation failed: client connection to 127.4.140.253:43893: connect: Connection refused (error 111)
I20260928 02:23:14.952793 27094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } attempt: 3
I20260928 02:23:14.954005 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:14.953535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:43893 (local address 127.0.0.1:60614)
0928 02:23:14.953744 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:14.953841 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 02:23:14.955142 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:14.954790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:43893 (local address 127.0.0.1:60628)
0928 02:23:14.954949 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:14.955016 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 02:23:14.955416 27094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.253:43893: Network error: Client connection negotiation failed: client connection to 127.4.140.253:43893: connect: Connection refused (error 111)
I20260928 02:23:15.098676 27094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } attempt: 4
I20260928 02:23:15.100143 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.099524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:43893 (local address 127.0.0.1:60630)
0928 02:23:15.099855 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.100005 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260928 02:23:15.101178 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.100856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:43893 (local address 127.0.0.1:60636)
0928 02:23:15.100983 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.101062 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:23:15.101449 27094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.253:43893: Network error: Client connection negotiation failed: client connection to 127.4.140.253:43893: connect: Connection refused (error 111)
W20260928 02:23:15.113531 27101 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:23:15.113870 27101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:23:15.113946 27101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:23:15.114029 27101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:23:15.121461 27101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:23:15.121564 27101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:23:15.121621 27101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:23:15.121672 27101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:23:15.121734 27101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:23:15.131798 27101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:45357,127.4.140.253:43893,127.4.140.252:38129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.253.keytab
--principal=kudu/127.4.140.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.253:43893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.140.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:23:15.133091 27101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:23:15.134351 27101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:23:15.146198 27106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:15.146219 27109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:15.146222 27107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:23:15.147203 27101 server_base.cc:1034] running on GCE node
I20260928 02:23:15.148056 27101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:23:15.149632 27101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:23:15.150851 27101 hybrid_clock.cc:648] HybridClock initialized: now 1790562195150792 us; error 29 us; skew 500 ppm
I20260928 02:23:15.151209 27101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:23:15 dist-test-slave-h37r krb5kdc[27014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562195, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:23:15.154413 27101 init.cc:377] Logged in from keytab as kudu/127.4.140.253@KRBTEST.COM (short username kudu)
I20260928 02:23:15.156006 27101 webserver.cc:460] Webserver started at http://127.4.140.253:37431/ using document root <none> and password file <none>
I20260928 02:23:15.156711 27101 fs_manager.cc:359] Metadata directory not provided
I20260928 02:23:15.156845 27101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:23:15.157186 27101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:23:15.160168 27101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-1/data/instance:
uuid: "2df2fa43f8c04f8892504ed9169de527"
format_stamp: "Formatted at 2026-09-28 02:23:15 on dist-test-slave-h37r"
server_key: "f4feec11582af1858c654e27c8a7bab3"
server_key_iv: "30358ef94c84d894efcb86746abf6e1e"
server_key_version: "encryptionkey@0"
I20260928 02:23:15.160991 27101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-1/wal/instance:
uuid: "2df2fa43f8c04f8892504ed9169de527"
format_stamp: "Formatted at 2026-09-28 02:23:15 on dist-test-slave-h37r"
server_key: "f4feec11582af1858c654e27c8a7bab3"
server_key_iv: "30358ef94c84d894efcb86746abf6e1e"
server_key_version: "encryptionkey@0"
I20260928 02:23:15.166750 27101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:23:15.170217 27116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:23:15.171571 27101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:23:15.171754 27101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-1/wal
uuid: "2df2fa43f8c04f8892504ed9169de527"
format_stamp: "Formatted at 2026-09-28 02:23:15 on dist-test-slave-h37r"
server_key: "f4feec11582af1858c654e27c8a7bab3"
server_key_iv: "30358ef94c84d894efcb86746abf6e1e"
server_key_version: "encryptionkey@0"
I20260928 02:23:15.172035 27101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:23:15.194170 27101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:23:15.198519 27101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:23:15.198822 27101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:23:15.218570 27101 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.253:43893
I20260928 02:23:15.218591 27167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.253:43893 every 8 connection(s)
I20260928 02:23:15.220007 27101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-1/data/info.pb
I20260928 02:23:15.223711 27168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:23:15.227128 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 27101
I20260928 02:23:15.227303 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-1/wal/instance
I20260928 02:23:15.227756 4659 external_mini_cluster.cc:1428] Setting key ded4c63b7200dbafa64f640de28d9099
I20260928 02:23:15.230770 27168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:23:15 dist-test-slave-h37r krb5kdc[27014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562195, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.253@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.140.252@KRBTEST.COM; defaulting to no policy
I20260928 02:23:15.251307 27170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.233642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.4.140.254:45357)
0928 02:23:15.234179 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:15.234183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:15.235202 (+ 1019us) server_negotiation.cc:407] Connection header received
0928 02:23:15.237147 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:15.237153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:15.237242 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:15.237371 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:15.238882 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.239624 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.241200 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.241612 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.241752 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.245228 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:15.245233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:15.245236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:15.245285 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:15.248074 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.248844 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.248852 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.248859 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.248984 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.249453 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.249464 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.249467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.249741 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:15.249920 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:15.250695 (+ 775us) server_negotiation.cc:299] Negotiation successful
0928 02:23:15.250925 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":319,"threads_started":1}
I20260928 02:23:15.252461 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.234295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:45357 (local address 127.0.0.1:53822)
0928 02:23:15.235033 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.235080 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:23:15.236875 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:23:15.237604 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:23:15.237612 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:23:15.238117 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:23:15.238665 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:15.238682 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.239789 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.239793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:15.241049 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:15.241061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.241858 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.241866 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:15.242092 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.242778 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:23:15.242829 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:23:15.244994 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:23:15.248215 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:15.248224 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:15.248242 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:15.248684 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:15.249097 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:15.249103 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:15.249109 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:15.249322 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:15.249889 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:23:15.249898 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:23:15.250335 (+ 437us) client_negotiation.cc:769] Sending connection context
0928 02:23:15.250609 (+ 274us) client_negotiation.cc:240] Negotiation successful
0928 02:23:15.251130 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":296,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.252@KRBTEST.COM" created.
I20260928 02:23:15.254513 27168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:23:15 dist-test-slave-h37r krb5kdc[27014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562195, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.253@KRBTEST.COM for kudu/127.4.140.253@KRBTEST.COM
I20260928 02:23:15.272693 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.255253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:43893 (local address 127.0.0.1:60644)
0928 02:23:15.255498 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.255519 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:23:15.255650 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:23:15.256931 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:23:15.256943 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:23:15.257449 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:23:15.257662 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:15.257675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.259277 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.259284 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:15.260895 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:15.260912 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.262119 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.262128 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:15.262295 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.262944 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:23:15.262970 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:23:15.265381 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:23:15.269876 (+ 4495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:15.269881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:15.269884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:15.270344 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:15.270851 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:15.270855 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:15.270857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:15.271017 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:15.271818 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:23:15.271821 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:23:15.272022 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 02:23:15.272144 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 02:23:15.272410 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 02:23:15.273181 27173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.255585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60644 (local address 127.4.140.253:43893)
0928 02:23:15.256491 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:15.256497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:15.256523 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:23:15.256567 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:15.256573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:15.256762 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:15.256882 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:15.257898 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.259115 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.261057 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.261944 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.262079 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.265605 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:15.265622 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:15.265644 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:15.265692 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:15.269701 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.270520 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.270532 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.270542 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.270700 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.271138 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.271146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.271152 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.271662 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:15.271931 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:15.272335 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 02:23:15.272634 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":534,"threads_started":1}
I20260928 02:23:15.275799 27168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:23:15.277192 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.276685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:51994)
0928 02:23:15.276917 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.277040 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 02:23:15.277547 27120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
I20260928 02:23:15.279351 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.278961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:52010)
0928 02:23:15.279119 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.279235 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 02:23:15.279691 27168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:38129: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.252.keytab.
Entry for principal kudu/127.4.140.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.252.keytab.
I20260928 02:23:15.292253 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.252:38129
--webserver_interface=127.4.140.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.252.keytab
--principal=kudu/127.4.140.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:45357,127.4.140.253:43893,127.4.140.252:38129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:23:15.329046 27168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } attempt: 1
I20260928 02:23:15.330384 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.329842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:52026)
0928 02:23:15.330101 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.330222 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260928 02:23:15.331506 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.331159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:52042)
0928 02:23:15.331315 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.331392 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 02:23:15.331758 27168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:38129: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
I20260928 02:23:15.399992 27168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } attempt: 2
I20260928 02:23:15.400710 27094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } attempt: 5
I20260928 02:23:15.401363 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.400776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:38400)
0928 02:23:15.401020 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.401142 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 02:23:15.402460 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.402064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:38404)
0928 02:23:15.402207 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.402297 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 02:23:15.402817 27168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:38129: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
Sep 28 02:23:15 dist-test-slave-h37r krb5kdc[27014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562194, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for kudu/127.4.140.253@KRBTEST.COM
I20260928 02:23:15.413718 27173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.401411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49310 (local address 127.4.140.253:43893)
0928 02:23:15.401672 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:15.401680 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:15.401709 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:23:15.401861 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:15.401879 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:15.401997 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:15.402125 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:15.403055 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.403747 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.404930 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.405373 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.405500 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.408515 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:15.408520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:15.408523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:15.408552 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:15.411097 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.411789 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.411796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.411800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.412003 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.412381 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.412386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.412390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.412665 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:15.412828 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:15.413177 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 02:23:15.413388 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 02:23:15.413734 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.401327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:43893 (local address 127.0.0.1:49310)
0928 02:23:15.401593 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.401612 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:23:15.401738 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:23:15.402179 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:23:15.402185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:23:15.402677 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:23:15.402906 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:15.402916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.403888 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.403924 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:15.404773 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:15.404784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.405523 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.405528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:15.405738 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.406340 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:23:15.406358 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:23:15.408333 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:23:15.411235 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:15.411241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:15.411245 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:15.411690 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:15.412113 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:15.412118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:15.412122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:15.412262 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:15.412782 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:23:15.412787 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:23:15.412938 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 02:23:15.413080 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 02:23:15.413393 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260928 02:23:15.414819 27094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:23:15.416172 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.415662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:38406)
0928 02:23:15.415878 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.416015 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 02:23:15.417201 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.416865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:38422)
0928 02:23:15.417017 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.417104 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 02:23:15.417465 27094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:38129: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
I20260928 02:23:15.468776 27094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } attempt: 1
I20260928 02:23:15.469969 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.469494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:38434)
0928 02:23:15.469691 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.469793 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 02:23:15.471074 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.470721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:38446)
0928 02:23:15.470876 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.470960 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 02:23:15.471305 27094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:38129: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
I20260928 02:23:15.494112 27168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } attempt: 3
I20260928 02:23:15.495411 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.494908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:38448)
0928 02:23:15.495140 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.495240 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 02:23:15.496559 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.496108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:38450)
0928 02:23:15.496364 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.496441 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260928 02:23:15.496833 27168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:38129: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
I20260928 02:23:15.539541 27094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } attempt: 2
I20260928 02:23:15.540786 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.540296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:38458)
0928 02:23:15.540523 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.540618 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 02:23:15.542053 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.541720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:38472)
0928 02:23:15.541864 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.541947 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 02:23:15.542309 27094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:38129: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
W20260928 02:23:15.635282 27176 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:23:15.635619 27176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:23:15.635782 27176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:23:15.635932 27176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 02:23:15.640075 27168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } attempt: 4
I20260928 02:23:15.641264 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.640800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:38474)
0928 02:23:15.641020 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.641111 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 02:23:15.642232 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.641908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:38478)
0928 02:23:15.642057 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.642133 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 02:23:15.642478 27168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:38129: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
W20260928 02:23:15.643857 27176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:23:15.644037 27176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:23:15.644156 27176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:23:15.644243 27176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:23:15.644345 27176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:23:15.648526 27094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } attempt: 3
I20260928 02:23:15.649576 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.649105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:38484)
0928 02:23:15.649335 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.649418 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 02:23:15.650631 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.650223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:38496)
0928 02:23:15.650455 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.650527 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260928 02:23:15.650854 27094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:38129: Network error: Client connection negotiation failed: client connection to 127.4.140.252:38129: connect: Connection refused (error 111)
I20260928 02:23:15.655323 27176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:45357,127.4.140.253:43893,127.4.140.252:38129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.252.keytab
--principal=kudu/127.4.140.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.252:38129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.140.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:23:15.656728 27176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:23:15.658048 27176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:23:15.670275 27181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:15.670253 27184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:15.670203 27182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:23:15.671303 27176 server_base.cc:1034] running on GCE node
I20260928 02:23:15.672165 27176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:23:15.673605 27176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:23:15.674852 27176 hybrid_clock.cc:648] HybridClock initialized: now 1790562195674779 us; error 46 us; skew 500 ppm
I20260928 02:23:15.675199 27176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:23:15 dist-test-slave-h37r krb5kdc[27014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562195, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:23:15.678535 27176 init.cc:377] Logged in from keytab as kudu/127.4.140.252@KRBTEST.COM (short username kudu)
I20260928 02:23:15.680002 27176 webserver.cc:460] Webserver started at http://127.4.140.252:39241/ using document root <none> and password file <none>
I20260928 02:23:15.680626 27176 fs_manager.cc:359] Metadata directory not provided
I20260928 02:23:15.680708 27176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:23:15.680972 27176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:23:15.683746 27176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-2/data/instance:
uuid: "79bac552a1814f3c803bad627208cc13"
format_stamp: "Formatted at 2026-09-28 02:23:15 on dist-test-slave-h37r"
server_key: "89cab9be356a08c29d561ef90f17970e"
server_key_iv: "6a33b39130554e762800b9dd02d51703"
server_key_version: "encryptionkey@0"
I20260928 02:23:15.684535 27176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-2/wal/instance:
uuid: "79bac552a1814f3c803bad627208cc13"
format_stamp: "Formatted at 2026-09-28 02:23:15 on dist-test-slave-h37r"
server_key: "89cab9be356a08c29d561ef90f17970e"
server_key_iv: "6a33b39130554e762800b9dd02d51703"
server_key_version: "encryptionkey@0"
I20260928 02:23:15.689445 27176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:23:15.692740 27191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:23:15.694056 27176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:23:15.694216 27176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-2/wal
uuid: "79bac552a1814f3c803bad627208cc13"
format_stamp: "Formatted at 2026-09-28 02:23:15 on dist-test-slave-h37r"
server_key: "89cab9be356a08c29d561ef90f17970e"
server_key_iv: "6a33b39130554e762800b9dd02d51703"
server_key_version: "encryptionkey@0"
I20260928 02:23:15.694447 27176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:23:15.716634 27176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:23:15.721024 27176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:23:15.721242 27176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:23:15.739537 27176 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.252:38129
I20260928 02:23:15.739559 27242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.252:38129 every 8 connection(s)
I20260928 02:23:15.740928 27176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-2/data/info.pb
I20260928 02:23:15.744854 27243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:23:15.748339 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 27176
I20260928 02:23:15.748555 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561835360118-4659-0/minicluster-data/master-2/wal/instance
I20260928 02:23:15.749097 4659 external_mini_cluster.cc:1428] Setting key a3e093941f4022e8b77c34d3253dbd24
I20260928 02:23:15.752956 27243 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:23:15 dist-test-slave-h37r krb5kdc[27014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.252@KRBTEST.COM
Sep 28 02:23:15 dist-test-slave-h37r krb5kdc[27014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562195, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.252@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:23:15.775166 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.756587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38556 (local address 127.4.140.254:45357)
0928 02:23:15.757151 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:15.757158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:15.757770 (+ 612us) server_negotiation.cc:407] Connection header received
0928 02:23:15.758012 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:15.758017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:15.758085 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:15.758208 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:15.759964 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.760987 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.763092 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.763522 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.763632 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.768745 (+ 5113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:15.768755 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:15.768761 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:15.768826 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:15.772028 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.772872 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.772877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.772880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.773024 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.773393 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.773400 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.773402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.773771 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:15.773914 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:15.774520 (+ 606us) server_negotiation.cc:299] Negotiation successful
0928 02:23:15.774733 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":289,"threads_started":1}
I20260928 02:23:15.775744 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.756663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:45357 (local address 127.0.0.1:38556)
0928 02:23:15.757594 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.757645 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:23:15.757867 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:23:15.758285 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:23:15.758292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:23:15.759268 (+ 976us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:23:15.759637 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:15.759651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.761220 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.761224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:15.762889 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:15.762906 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.763791 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.763800 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:15.764063 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.765121 (+ 1058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:23:15.765146 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:23:15.768484 (+ 3338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:23:15.772184 (+ 3700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:15.772196 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:15.772203 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:15.772747 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:15.773133 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:15.773136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:15.773138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:15.773256 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:15.773911 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:23:15.773919 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:23:15.774141 (+ 222us) client_negotiation.cc:769] Sending connection context
0928 02:23:15.774392 (+ 251us) client_negotiation.cc:240] Negotiation successful
0928 02:23:15.774708 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":391,"threads_started":1}
I20260928 02:23:15.775713 27246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.751162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.4.140.252:38129)
0928 02:23:15.752028 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:15.752040 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:15.752081 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:23:15.753654 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:15.753677 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:15.754107 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:15.754515 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:15.756352 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.758470 (+ 2118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.760291 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.761363 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.761506 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.766847 (+ 5341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:15.766865 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:15.766887 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:15.766933 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:15.770993 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.771815 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.771837 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.771849 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.772083 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.772523 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.772526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.772528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.773294 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:15.773611 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:15.774057 (+ 446us) server_negotiation.cc:299] Negotiation successful
0928 02:23:15.774481 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"spinlock_wait_cycles":39296,"thread_start_us":411,"threads_started":1}
I20260928 02:23:15.777582 27243 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:23:15 dist-test-slave-h37r krb5kdc[27014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562195, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.252@KRBTEST.COM for kudu/127.4.140.253@KRBTEST.COM
Sep 28 02:23:15 dist-test-slave-h37r krb5kdc[27014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:23:15.792716 27173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.778402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49318 (local address 127.4.140.253:43893)
0928 02:23:15.778600 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:15.778606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:15.778627 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:23:15.778833 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:15.778839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:15.778942 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:15.779045 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:15.780114 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.780843 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.782212 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.782630 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.782728 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.786561 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:15.786567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:15.786570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:15.786604 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:15.790065 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.790737 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.790745 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.790748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.790939 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.791302 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.791307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.791311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.791664 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:15.791851 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:15.792258 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 02:23:15.792452 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:23:15.792793 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.778221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:43893 (local address 127.0.0.1:49318)
0928 02:23:15.778551 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.778566 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:23:15.778695 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:23:15.779240 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:23:15.779246 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:23:15.779731 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:23:15.779960 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:15.779976 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.780991 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.780995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:15.782063 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:15.782078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.782779 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.782786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:15.783009 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.783634 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:23:15.783651 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:23:15.786345 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:23:15.790211 (+ 3866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:15.790215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:15.790217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:15.790607 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:15.791056 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:15.791059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:15.791061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:15.791193 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:15.791798 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:23:15.791801 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:23:15.792008 (+ 207us) client_negotiation.cc:769] Sending connection context
0928 02:23:15.792122 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:23:15.792422 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239}
I20260928 02:23:15.793800 27243 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:23:15.795193 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.778947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38558 (local address 127.4.140.254:45357)
0928 02:23:15.779104 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:15.779108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:15.779125 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:23:15.779338 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:15.779341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:15.779405 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:15.779478 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:15.780843 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.781552 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.783021 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.783646 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.783738 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.788620 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:15.788627 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:15.788633 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:15.788678 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:15.792153 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.793017 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.793026 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.793030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.793185 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.793632 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.793637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.793640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.793948 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:15.794198 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:15.794602 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 02:23:15.794844 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 02:23:15 dist-test-slave-h37r krb5kdc[27014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562195, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.252@KRBTEST.COM for kudu/127.4.140.252@KRBTEST.COM
Sep 28 02:23:15 dist-test-slave-h37r krb5kdc[27014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.253@KRBTEST.COM
I20260928 02:23:15.808126 27246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.794609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38514 (local address 127.4.140.252:38129)
0928 02:23:15.794804 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:15.794810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:15.795112 (+ 302us) server_negotiation.cc:407] Connection header received
0928 02:23:15.795235 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:15.795239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:15.795352 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:15.795452 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:15.796222 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.797153 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.798446 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.799078 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.799199 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.802448 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:15.802454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:15.802460 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:15.802501 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:15.805624 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.806236 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.806245 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.806248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.806378 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.806735 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.806742 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.806746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.807010 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:15.807154 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:15.807493 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 02:23:15.807735 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":58}
I20260928 02:23:15.808246 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.794703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:38514)
0928 02:23:15.795016 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.795035 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:23:15.795141 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:23:15.795500 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:23:15.795504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:23:15.795832 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:23:15.796065 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:15.796079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.797305 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.797310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:15.798305 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:15.798316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.799202 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.799206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:15.799464 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.800131 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:23:15.800152 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:23:15.802230 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:23:15.805768 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:15.805772 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:15.805776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:15.806121 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:15.806482 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:15.806488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:15.806509 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:15.806635 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:15.807131 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:23:15.807136 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:23:15.807274 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 02:23:15.807421 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 02:23:15.807879 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":153}
I20260928 02:23:15.812449 27173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.797171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49324 (local address 127.4.140.253:43893)
0928 02:23:15.797334 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:15.797340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:15.797360 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:23:15.797441 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:15.797446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:15.797511 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:15.797606 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:15.798878 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.799736 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.801045 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.801489 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.801633 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.806645 (+ 5012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:15.806650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:15.806654 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:15.806686 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:15.809696 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.810344 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.810352 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.810358 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.810541 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.810919 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.810924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.810927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.811285 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:15.811567 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:15.811778 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 02:23:15.812079 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:23:15.814611 27243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13: Bootstrap starting.
I20260928 02:23:15.817942 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:23:15.818269 27243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13: Neither blocks nor log segments found. Creating new log.
I20260928 02:23:15.819351 27243 log.cc:824] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13: Log is configured to *not* fsync() on all Append() calls
I20260928 02:23:15.822270 27243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13: No bootstrap required, opened a new log
I20260928 02:23:15.820803 27069 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fb9ab4df0efd456f906090a08889d9ee"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38558
I20260928 02:23:15.828084 27094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } attempt: 4
I20260928 02:23:15.830102 27243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9ab4df0efd456f906090a08889d9ee" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } } peers { permanent_uuid: "2df2fa43f8c04f8892504ed9169de527" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } } peers { permanent_uuid: "79bac552a1814f3c803bad627208cc13" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } }
I20260928 02:23:15.830523 27243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79bac552a1814f3c803bad627208cc13, State: Initialized, Role: FOLLOWER
I20260928 02:23:15.831079 27243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9ab4df0efd456f906090a08889d9ee" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } } peers { permanent_uuid: "2df2fa43f8c04f8892504ed9169de527" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } } peers { permanent_uuid: "79bac552a1814f3c803bad627208cc13" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } }
I20260928 02:23:15.832180 27250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9ab4df0efd456f906090a08889d9ee" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } } peers { permanent_uuid: "2df2fa43f8c04f8892504ed9169de527" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } } peers { permanent_uuid: "79bac552a1814f3c803bad627208cc13" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } } }
I20260928 02:23:15.832564 27250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:23:15.833446 27243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 02:23:15 dist-test-slave-h37r krb5kdc[27014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562194, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for kudu/127.4.140.252@KRBTEST.COM
W20260928 02:23:15.845350 27261 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 02:23:15.845506 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.828775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:38520)
0928 02:23:15.829034 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.829056 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:23:15.829192 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:23:15.829615 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:23:15.829620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:23:15.830015 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:23:15.830173 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:15.830181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.831372 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.831378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:15.832915 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:15.832928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.833797 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.833804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:15.834006 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.834686 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:23:15.834711 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:23:15.837611 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:23:15.842290 (+ 4679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:15.842299 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:15.842304 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:15.842877 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:15.843408 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:15.843412 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:15.843416 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:15.843581 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:15.844436 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:23:15.844444 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:23:15.844628 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 02:23:15.844762 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 02:23:15.845064 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
W20260928 02:23:15.845510 27261 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:23:15.846860 27246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.828828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38520 (local address 127.4.140.252:38129)
0928 02:23:15.829008 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:15.829013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:15.829135 (+ 122us) server_negotiation.cc:407] Connection header received
0928 02:23:15.829323 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:15.829329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:15.829448 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:15.829562 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:15.830329 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.831204 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.833078 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.833605 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.833790 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.837933 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:15.837941 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:15.837948 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:15.838003 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:15.842126 (+ 4123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.843062 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.843072 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.843076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.843249 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.843713 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.843718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.843721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.844258 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:15.846075 (+ 1817us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:15.846218 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 02:23:15.846485 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:23:15.853024 27094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee: Bootstrap starting.
I20260928 02:23:15.842507 27176 master_runner.cc:186] Error getting master registration for 127.4.140.254:45357: OK, instance_id { permanent_uuid: "fb9ab4df0efd456f906090a08889d9ee" instance_seqno: 1790562194689635 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:23:15.853585 27176 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:23:15.857446 27094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee: Neither blocks nor log segments found. Creating new log.
I20260928 02:23:15.858747 27094 log.cc:824] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee: Log is configured to *not* fsync() on all Append() calls
I20260928 02:23:15.862129 27094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee: No bootstrap required, opened a new log
I20260928 02:23:15.868212 27094 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9ab4df0efd456f906090a08889d9ee" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } } peers { permanent_uuid: "2df2fa43f8c04f8892504ed9169de527" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } } peers { permanent_uuid: "79bac552a1814f3c803bad627208cc13" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } }
I20260928 02:23:15.868587 27094 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb9ab4df0efd456f906090a08889d9ee, State: Initialized, Role: FOLLOWER
I20260928 02:23:15.869119 27094 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9ab4df0efd456f906090a08889d9ee" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } } peers { permanent_uuid: "2df2fa43f8c04f8892504ed9169de527" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } } peers { permanent_uuid: "79bac552a1814f3c803bad627208cc13" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } }
I20260928 02:23:15.870131 27264 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9ab4df0efd456f906090a08889d9ee" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } } peers { permanent_uuid: "2df2fa43f8c04f8892504ed9169de527" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } } peers { permanent_uuid: "79bac552a1814f3c803bad627208cc13" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } } }
I20260928 02:23:15.870514 27264 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:23:15.871308 27094 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:23:15.871415 27069 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 02:23:15.871632 27069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:23:15.875232 27069 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9ab4df0efd456f906090a08889d9ee" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } } peers { permanent_uuid: "2df2fa43f8c04f8892504ed9169de527" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } } peers { permanent_uuid: "79bac552a1814f3c803bad627208cc13" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } }
I20260928 02:23:15.878155 27218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb9ab4df0efd456f906090a08889d9ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "79bac552a1814f3c803bad627208cc13"
I20260928 02:23:15.878757 27218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:23:15.882409 27218 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb9ab4df0efd456f906090a08889d9ee in term 1.
I20260928 02:23:15.883127 27046 leader_election.cc:302] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79bac552a1814f3c803bad627208cc13, fb9ab4df0efd456f906090a08889d9ee; no voters:
I20260928 02:23:15.883790 27069 leader_election.cc:288] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [CANDIDATE]: Term 1 election: Requested vote from peers 2df2fa43f8c04f8892504ed9169de527 (127.4.140.253:43893), 79bac552a1814f3c803bad627208cc13 (127.4.140.252:38129)
I20260928 02:23:15.883761 27264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:23:15.897331 27264 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [term 1 LEADER]: Becoming Leader. State: Replica: fb9ab4df0efd456f906090a08889d9ee, State: Running, Role: LEADER
I20260928 02:23:15.899206 27264 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9ab4df0efd456f906090a08889d9ee" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } } peers { permanent_uuid: "2df2fa43f8c04f8892504ed9169de527" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } } peers { permanent_uuid: "79bac552a1814f3c803bad627208cc13" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } }
I20260928 02:23:15.922541 27272 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb9ab4df0efd456f906090a08889d9ee. Latest consensus state: current_term: 1 leader_uuid: "fb9ab4df0efd456f906090a08889d9ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9ab4df0efd456f906090a08889d9ee" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } } peers { permanent_uuid: "2df2fa43f8c04f8892504ed9169de527" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } } peers { permanent_uuid: "79bac552a1814f3c803bad627208cc13" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } } }
I20260928 02:23:15.923120 27272 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [sys.catalog]: This master's current role is: LEADER
W20260928 02:23:15.929586 27048 tablet.cc:2369] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:23:15.930531 27284 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:23:15.933972 27283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:23:15.941682 27168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } attempt: 5
I20260928 02:23:15.951663 27283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:23:15.935282 27027 master_runner.cc:186] Error getting master registration for 127.4.140.253:43893: OK, instance_id { permanent_uuid: "2df2fa43f8c04f8892504ed9169de527" instance_seqno: 1790562195213056 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:23:15.954278 27027 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:23:15.957521 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.912390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38562 (local address 127.4.140.254:45357)
0928 02:23:15.912582 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:15.912589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:15.919689 (+ 7100us) server_negotiation.cc:407] Connection header received
0928 02:23:15.920238 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:15.920245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:15.920335 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:15.920412 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:15.927707 (+ 7295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.928998 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.935870 (+ 6872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.936686 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.936792 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.944605 (+ 7813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:15.944615 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:15.944623 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:15.944701 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:15.949579 (+ 4878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.950710 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.950720 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.950726 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.950966 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.951526 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.951531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.951533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.951955 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:15.952264 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:15.954818 (+ 2554us) server_negotiation.cc:299] Negotiation successful
0928 02:23:15.957032 (+ 2214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:23:15.963488 27246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.913675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38522 (local address 127.4.140.252:38129)
0928 02:23:15.913850 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:15.913856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:15.923519 (+ 9663us) server_negotiation.cc:407] Connection header received
0928 02:23:15.924821 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:15.924828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:15.924950 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:15.925033 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:15.932778 (+ 7745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.933899 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.940210 (+ 6311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.940871 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.940968 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.952845 (+ 11877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:15.952852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:15.952857 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:15.952902 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:15.956603 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.960862 (+ 4259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.960870 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.960877 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.961066 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.961528 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.961534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.961539 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.961946 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:15.962241 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:15.962820 (+ 579us) server_negotiation.cc:299] Negotiation successful
0928 02:23:15.963083 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 02:23:15 dist-test-slave-h37r krb5kdc[27014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562195, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.253@KRBTEST.COM for kudu/127.4.140.252@KRBTEST.COM
I20260928 02:23:15.968820 27173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.924094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49328 (local address 127.4.140.253:43893)
0928 02:23:15.924285 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:15.924291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:15.931493 (+ 7202us) server_negotiation.cc:407] Connection header received
0928 02:23:15.931759 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:15.931766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:15.931849 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:15.931952 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:15.940034 (+ 8082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.941116 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.951696 (+ 10580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.952449 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.952551 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.956320 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:15.956328 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:15.956333 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:15.956378 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:15.960245 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.965251 (+ 5006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.965258 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.965261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.965431 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.965954 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.965963 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.965969 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.966715 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:15.967155 (+ 440us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:15.967936 (+ 781us) server_negotiation.cc:299] Negotiation successful
0928 02:23:15.968314 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 02:23:15.973233 27197 tablet.cc:2369] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:23:15.976094 27218 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13 [term 1 FOLLOWER]: Refusing update from remote peer fb9ab4df0efd456f906090a08889d9ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:23:15.977267 27264 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "79bac552a1814f3c803bad627208cc13" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:23:15.983775 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.942250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:38129 (local address 127.0.0.1:38526)
0928 02:23:15.942609 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:15.942628 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:23:15.942731 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:23:15.954451 (+ 11720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:23:15.954456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:23:15.955003 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:23:15.955221 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:15.955233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.959241 (+ 4008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.959248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:15.960963 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:15.960978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.962757 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.962764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:15.963028 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.963701 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:23:15.963725 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:23:15.967116 (+ 3391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:23:15.979957 (+ 12841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:15.979965 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:15.979971 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:15.980571 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:15.981281 (+ 710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:15.981288 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:15.981292 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:15.981474 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:15.982707 (+ 1233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:23:15.982712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:23:15.982887 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 02:23:15.983032 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 02:23:15.983360 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225}
I20260928 02:23:15.986155 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:15.942970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38526 (local address 127.4.140.252:38129)
0928 02:23:15.953154 (+ 10184us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:15.953165 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:15.953298 (+ 133us) server_negotiation.cc:407] Connection header received
0928 02:23:15.953628 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:15.953635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:15.954023 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:15.955607 (+ 1584us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:15.956226 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.958982 (+ 2756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.961254 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:15.962507 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:15.963397 (+ 890us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:15.975627 (+ 12230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:15.975634 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:15.975639 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:15.975685 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:15.979782 (+ 4097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.980746 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.980753 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.980757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.981082 (+ 325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:15.981657 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:15.981662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:15.981666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:15.982479 (+ 813us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:15.984201 (+ 1722us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:15.984560 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 02:23:15.985231 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9457,"thread_start_us":8494,"threads_started":1}
I20260928 02:23:15.992188 27168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2df2fa43f8c04f8892504ed9169de527: Bootstrap starting.
I20260928 02:23:15.996986 27168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2df2fa43f8c04f8892504ed9169de527: Neither blocks nor log segments found. Creating new log.
I20260928 02:23:15.999521 27168 log.cc:824] T 00000000000000000000000000000000 P 2df2fa43f8c04f8892504ed9169de527: Log is configured to *not* fsync() on all Append() calls
I20260928 02:23:16.006381 27168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2df2fa43f8c04f8892504ed9169de527: No bootstrap required, opened a new log
I20260928 02:23:16.008317 27250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb9ab4df0efd456f906090a08889d9ee. Latest consensus state: current_term: 1 leader_uuid: "fb9ab4df0efd456f906090a08889d9ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9ab4df0efd456f906090a08889d9ee" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } } peers { permanent_uuid: "2df2fa43f8c04f8892504ed9169de527" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } } peers { permanent_uuid: "79bac552a1814f3c803bad627208cc13" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } } }
I20260928 02:23:16.008649 27250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:23:16.017987 27168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2df2fa43f8c04f8892504ed9169de527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9ab4df0efd456f906090a08889d9ee" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } } peers { permanent_uuid: "2df2fa43f8c04f8892504ed9169de527" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } } peers { permanent_uuid: "79bac552a1814f3c803bad627208cc13" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } }
I20260928 02:23:16.018435 27168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2df2fa43f8c04f8892504ed9169de527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df2fa43f8c04f8892504ed9169de527, State: Initialized, Role: FOLLOWER
I20260928 02:23:16.018976 27168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2df2fa43f8c04f8892504ed9169de527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9ab4df0efd456f906090a08889d9ee" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } } peers { permanent_uuid: "2df2fa43f8c04f8892504ed9169de527" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } } peers { permanent_uuid: "79bac552a1814f3c803bad627208cc13" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } }
I20260928 02:23:16.027297 27289 mvcc.cc:205] Tried to move back new op lower bound from 7334142754702987264 to 7334142754445848576. Current Snapshot: MvccSnapshot[applied={T|T < 7334142754702987264}]
I20260928 02:23:16.032580 27294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2df2fa43f8c04f8892504ed9169de527 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9ab4df0efd456f906090a08889d9ee" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } } peers { permanent_uuid: "2df2fa43f8c04f8892504ed9169de527" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } } peers { permanent_uuid: "79bac552a1814f3c803bad627208cc13" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } } }
I20260928 02:23:16.033087 27294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2df2fa43f8c04f8892504ed9169de527 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:23:16.034076 27168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2df2fa43f8c04f8892504ed9169de527 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:23:16.034535 27142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2df2fa43f8c04f8892504ed9169de527 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:23:16.039496 27142 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2df2fa43f8c04f8892504ed9169de527 [term 1 FOLLOWER]: Refusing update from remote peer fb9ab4df0efd456f906090a08889d9ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:23:16.034099 27143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb9ab4df0efd456f906090a08889d9ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2df2fa43f8c04f8892504ed9169de527"
I20260928 02:23:16.042239 27250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb9ab4df0efd456f906090a08889d9ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9ab4df0efd456f906090a08889d9ee" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } } peers { permanent_uuid: "2df2fa43f8c04f8892504ed9169de527" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } } peers { permanent_uuid: "79bac552a1814f3c803bad627208cc13" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } } }
I20260928 02:23:16.042505 27250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:23:16.044910 27143 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2df2fa43f8c04f8892504ed9169de527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb9ab4df0efd456f906090a08889d9ee in term 1.
I20260928 02:23:16.046241 27272 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df2fa43f8c04f8892504ed9169de527" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:23:16.056789 27283 catalog_manager.cc:1347] Generated new cluster ID: c981d6890f82430183ba9359e41b7821
I20260928 02:23:16.056923 27283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:23:16.058749 27272 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb9ab4df0efd456f906090a08889d9ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9ab4df0efd456f906090a08889d9ee" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } } peers { permanent_uuid: "2df2fa43f8c04f8892504ed9169de527" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } } peers { permanent_uuid: "79bac552a1814f3c803bad627208cc13" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } } }
I20260928 02:23:16.059010 27272 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [sys.catalog]: This master's current role is: LEADER
I20260928 02:23:16.086724 27294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2df2fa43f8c04f8892504ed9169de527 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb9ab4df0efd456f906090a08889d9ee. Latest consensus state: current_term: 1 leader_uuid: "fb9ab4df0efd456f906090a08889d9ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9ab4df0efd456f906090a08889d9ee" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } } peers { permanent_uuid: "2df2fa43f8c04f8892504ed9169de527" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } } peers { permanent_uuid: "79bac552a1814f3c803bad627208cc13" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } } }
I20260928 02:23:16.087086 27294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2df2fa43f8c04f8892504ed9169de527 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:23:16.087792 27272 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb9ab4df0efd456f906090a08889d9ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9ab4df0efd456f906090a08889d9ee" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } } peers { permanent_uuid: "2df2fa43f8c04f8892504ed9169de527" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } } peers { permanent_uuid: "79bac552a1814f3c803bad627208cc13" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } } }
I20260928 02:23:16.088604 27272 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee [sys.catalog]: This master's current role is: LEADER
I20260928 02:23:16.090808 27283 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:23:16.092651 27283 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:23:16.100907 27301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2df2fa43f8c04f8892504ed9169de527 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb9ab4df0efd456f906090a08889d9ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9ab4df0efd456f906090a08889d9ee" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 45357 } } peers { permanent_uuid: "2df2fa43f8c04f8892504ed9169de527" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 43893 } } peers { permanent_uuid: "79bac552a1814f3c803bad627208cc13" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 38129 } } }
I20260928 02:23:16.101424 27301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2df2fa43f8c04f8892504ed9169de527 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:23:16.113318 27283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb9ab4df0efd456f906090a08889d9ee: Generated new TSK 0
I20260928 02:23:16.114512 27283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:23:16.128638 27309 catalog_manager.cc:1259] Loaded cluster ID: c981d6890f82430183ba9359e41b7821
I20260928 02:23:16.128834 27309 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2df2fa43f8c04f8892504ed9169de527: loading cluster ID for follower catalog manager: success
I20260928 02:23:16.132699 27309 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2df2fa43f8c04f8892504ed9169de527: acquiring CA information for follower catalog manager: success
I20260928 02:23:16.135644 27309 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2df2fa43f8c04f8892504ed9169de527: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 02:23:16.251096 27247 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:16.238378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38568 (local address 127.4.140.254:45357)
0928 02:23:16.238610 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.238617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.247558 (+ 8941us) server_negotiation.cc:407] Connection header received
0928 02:23:16.248745 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.248754 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.248876 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.248958 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:16.250830 (+ 1872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38568: BlockingRecv error: recv got EOF from 127.0.0.1:38568 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 02:23:16.255367 27173 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:16.252938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49334 (local address 127.4.140.253:43893)
0928 02:23:16.253109 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.253116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.253751 (+ 635us) server_negotiation.cc:407] Connection header received
0928 02:23:16.254053 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.254059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.254155 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.254224 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:16.255129 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49334: BlockingRecv error: recv got EOF from 127.0.0.1:49334 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 02:23:16.257038 27287 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:16.242081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.4.140.252:38129)
0928 02:23:16.242281 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.242288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.252318 (+ 10030us) server_negotiation.cc:407] Connection header received
0928 02:23:16.252477 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.252483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.252593 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.252701 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:16.256796 (+ 4095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38530: BlockingRecv error: recv got EOF from 127.0.0.1:38530 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:23:16.316804 27173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:16.299037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49344 (local address 127.4.140.253:43893)
0928 02:23:16.299258 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.299265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.307193 (+ 7928us) server_negotiation.cc:407] Connection header received
0928 02:23:16.307785 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.307792 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.308057 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.308205 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:16.309024 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.310227 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.312353 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.314225 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.314333 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:16.315166 (+ 833us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:16.315915 (+ 749us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:16.316044 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:16.316300 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 02:23:16.316405 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 02:23:16.374071 27287 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:16.357680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38534 (local address 127.4.140.252:38129)
0928 02:23:16.357879 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.357885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.372041 (+ 14156us) server_negotiation.cc:407] Connection header received
0928 02:23:16.372410 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.372418 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.372563 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.372649 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:16.373773 (+ 1124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38534: BlockingRecv error: recv got EOF from 127.0.0.1:38534 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:23:16.430352 27173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:16.413335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49354 (local address 127.4.140.253:43893)
0928 02:23:16.413531 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.413537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.421526 (+ 7989us) server_negotiation.cc:407] Connection header received
0928 02:23:16.422091 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.422099 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.422228 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.422301 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:16.423078 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.424305 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.426404 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.428609 (+ 2205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.428719 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:16.429424 (+ 705us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:16.429601 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:16.429666 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:16.429932 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 02:23:16.430031 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 02:23:16.482982 27287 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:16.464652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38546 (local address 127.4.140.252:38129)
0928 02:23:16.464865 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.464872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.481325 (+ 16453us) server_negotiation.cc:407] Connection header received
0928 02:23:16.481540 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.481547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.481637 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.481715 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:16.482750 (+ 1035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38546: BlockingRecv error: recv got EOF from 127.0.0.1:38546 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 02:23:16.531803 27173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:16.512709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.4.140.253:43893)
0928 02:23:16.512903 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.512910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.524689 (+ 11779us) server_negotiation.cc:407] Connection header received
0928 02:23:16.524915 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.524924 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.525029 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.525157 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:16.525685 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.526964 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.529005 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.529755 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.529901 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:16.530405 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:16.530576 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:16.530679 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:16.531096 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 02:23:16.531208 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 02:23:16.546213 27287 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:16.539853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38558 (local address 127.4.140.252:38129)
0928 02:23:16.540070 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.540076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.544549 (+ 4473us) server_negotiation.cc:407] Connection header received
0928 02:23:16.544780 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.544786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.544900 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.545024 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:16.545933 (+ 909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38558: BlockingRecv error: recv got EOF from 127.0.0.1:38558 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 02:23:16.575073 27173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:16.564120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49364 (local address 127.4.140.253:43893)
0928 02:23:16.564318 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.564324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.568512 (+ 4188us) server_negotiation.cc:407] Connection header received
0928 02:23:16.568741 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.568748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.568847 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.568958 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:16.569495 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.570678 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.572761 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.573530 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.573670 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:16.574208 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:16.574369 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:16.574479 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:16.574655 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 02:23:16.574749 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 02:23:16.591562 27287 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:16.589077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38560 (local address 127.4.140.252:38129)
0928 02:23:16.589356 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.589363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.589969 (+ 606us) server_negotiation.cc:407] Connection header received
0928 02:23:16.590213 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.590219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.590312 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.590411 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:16.591339 (+ 928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38560: BlockingRecv error: recv got EOF from 127.0.0.1:38560 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
I20260928 02:23:16.611775 27173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:16.604278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49380 (local address 127.4.140.253:43893)
0928 02:23:16.604487 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.604494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.605028 (+ 534us) server_negotiation.cc:407] Connection header received
0928 02:23:16.605377 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.605386 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.605476 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.605579 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:16.606098 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.607300 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.609315 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.610159 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.610290 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:16.610810 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:16.610955 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:16.611041 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:16.611273 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 02:23:16.611364 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 02:23:16.621868 27287 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:16.619330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38562 (local address 127.4.140.252:38129)
0928 02:23:16.619543 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.619549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.620312 (+ 763us) server_negotiation.cc:407] Connection header received
0928 02:23:16.620576 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.620583 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.620677 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.620784 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:16.621643 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38562: BlockingRecv error: recv got EOF from 127.0.0.1:38562 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 02:23:16.647658 27173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:16.641161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49388 (local address 127.4.140.253:43893)
0928 02:23:16.641372 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.641380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.641968 (+ 588us) server_negotiation.cc:407] Connection header received
0928 02:23:16.642230 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.642235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.642314 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.642427 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:16.642956 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.643811 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.645841 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.646285 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.646406 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:16.646898 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:16.647031 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:16.647100 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:16.647273 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 02:23:16.647347 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 02:23:16.656215 27287 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:16.654118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.4.140.252:38129)
0928 02:23:16.654362 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.654369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.654713 (+ 344us) server_negotiation.cc:407] Connection header received
0928 02:23:16.654968 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.654973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.655058 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.655172 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:16.655972 (+ 800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38572: BlockingRecv error: recv got EOF from 127.0.0.1:38572 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 02:23:16.697371 27173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:16.691693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49390 (local address 127.4.140.253:43893)
0928 02:23:16.691965 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.691972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.692346 (+ 374us) server_negotiation.cc:407] Connection header received
0928 02:23:16.692554 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.692560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.692651 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.692758 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:16.693158 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.693983 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.695289 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.696000 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.696117 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:16.696602 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:16.696764 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:16.696856 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:16.696987 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 02:23:16.697068 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 02:23:16.710072 27287 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:16.703499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38578 (local address 127.4.140.252:38129)
0928 02:23:16.703742 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.703752 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.708532 (+ 4780us) server_negotiation.cc:407] Connection header received
0928 02:23:16.708728 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.708732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.708826 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.708924 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:16.709819 (+ 895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38578: BlockingRecv error: recv got EOF from 127.0.0.1:38578 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 02:23:16.783619 27173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:16.778040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.4.140.253:43893)
0928 02:23:16.778308 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.778316 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.778639 (+ 323us) server_negotiation.cc:407] Connection header received
0928 02:23:16.778912 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.778916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.778990 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.779092 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:16.779532 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.780422 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.781833 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.782273 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.782421 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:16.782850 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:16.782973 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:16.783077 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:16.783229 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 02:23:16.783314 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260928 02:23:16.791778 27287 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:16.789556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.4.140.252:38129)
0928 02:23:16.789904 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.789911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.790185 (+ 274us) server_negotiation.cc:407] Connection header received
0928 02:23:16.790428 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.790440 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.790514 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.790665 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:16.791576 (+ 911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38592: BlockingRecv error: recv got EOF from 127.0.0.1:38592 (error 108)
Metrics: {"server-negotiator.queue_time_us":180}
I20260928 02:23:16.847702 27261 catalog_manager.cc:1259] Loaded cluster ID: c981d6890f82430183ba9359e41b7821
I20260928 02:23:16.847860 27261 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13: loading cluster ID for follower catalog manager: success
I20260928 02:23:16.850917 27261 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13: acquiring CA information for follower catalog manager: success
I20260928 02:23:16.852855 27261 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 79bac552a1814f3c803bad627208cc13: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:23:16.928802 27173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:16.923874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.4.140.253:43893)
0928 02:23:16.924122 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.924126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.924571 (+ 445us) server_negotiation.cc:407] Connection header received
0928 02:23:16.924772 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.924777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.924863 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.924956 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:16.925352 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.926064 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.927360 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.927758 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.927850 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:16.928239 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:16.928342 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:16.928406 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:16.928532 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:23:16.928592 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260928 02:23:16.941138 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:16.934919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38594 (local address 127.4.140.252:38129)
0928 02:23:16.935125 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.935133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.935469 (+ 336us) server_negotiation.cc:407] Connection header received
0928 02:23:16.935693 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.935697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.935862 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.936040 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:16.936483 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.937427 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.938848 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.939280 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.939420 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:16.939990 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:16.940517 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:16.940688 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:16.940755 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 02:23:16.940844 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:23:16.955497 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:16.947717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.4.140.252:38129)
0928 02:23:16.947982 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.947988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.948731 (+ 743us) server_negotiation.cc:407] Connection header received
0928 02:23:16.949016 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.949073 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.949159 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.949294 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:16.949954 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.950776 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.953331 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.953937 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.954087 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:16.954655 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:16.954826 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:16.954935 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:16.955101 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 02:23:16.955184 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:23:16.960276 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:16.947504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38582 (local address 127.4.140.254:45357)
0928 02:23:16.948125 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.948134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.953101 (+ 4967us) server_negotiation.cc:407] Connection header received
0928 02:23:16.953250 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.953259 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.953527 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.953719 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:16.954802 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.955967 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.957837 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.958264 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.958406 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:16.959013 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:16.959555 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:16.959680 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:16.959806 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:23:16.959933 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":250,"threads_started":1}
I20260928 02:23:16.961089 27173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:16.952876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49418 (local address 127.4.140.253:43893)
0928 02:23:16.953079 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:16.953084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:16.953343 (+ 259us) server_negotiation.cc:407] Connection header received
0928 02:23:16.953540 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:16.953546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:16.953644 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:16.953725 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:16.954756 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.955955 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.957823 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:16.958329 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:16.958483 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:16.960273 (+ 1790us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:16.960420 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:16.960510 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:16.960705 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 02:23:16.960799 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:23:16.966550 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 27027
I20260928 02:23:16.967650 27089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:23:17.079655 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 27027
I20260928 02:23:17.101279 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 27101
I20260928 02:23:17.102743 27163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:23:17.213547 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 27101
I20260928 02:23:17.234324 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 27176
I20260928 02:23:17.235478 27238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:23:17.346681 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 27176
2026-09-28T02:23:17Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4487 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:23:17 dist-test-slave-h37r krb5kdc[27423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:23:17 dist-test-slave-h37r krb5kdc[27423](info): set up 2 sockets
Sep 28 02:23:17 dist-test-slave-h37r krb5kdc[27423](info): commencing operation
krb5kdc: starting...
W20260928 02:23:18.468432 4659 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/test-user.keytab.
Sep 28 02:23:18 dist-test-slave-h37r krb5kdc[27423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:23:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:23:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.140.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
Entry for principal kudu/127.4.140.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab.
I20260928 02:23:18.747870 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:42211
--webserver_interface=127.4.140.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:42365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:42211,127.4.140.253:38709,127.4.140.252:42101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:23:19.091049 27436 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:23:19.091503 27436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:23:19.091641 27436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:23:19.091794 27436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:23:19.099310 27436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:23:19.099453 27436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:23:19.099560 27436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:23:19.099651 27436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:23:19.099750 27436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:23:19.110100 27436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:42365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:42211,127.4.140.253:38709,127.4.140.252:42101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.254.keytab
--principal=kudu/127.4.140.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.254:42211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.140.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:23:19.111539 27436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:23:19.112929 27436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:23:19.125041 27442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:19.125044 27441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:19.125798 27444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:23:19.126124 27436 server_base.cc:1034] running on GCE node
I20260928 02:23:19.126912 27436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:23:19.128518 27436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:23:19.129741 27436 hybrid_clock.cc:648] HybridClock initialized: now 1790562199129655 us; error 56 us; skew 500 ppm
I20260928 02:23:19.130095 27436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:23:19 dist-test-slave-h37r krb5kdc[27423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562199, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:23:19.144083 27436 init.cc:377] Logged in from keytab as kudu/127.4.140.254@KRBTEST.COM (short username kudu)
I20260928 02:23:19.145735 27436 webserver.cc:460] Webserver started at http://127.4.140.254:45435/ using document root <none> and password file <none>
I20260928 02:23:19.146508 27436 fs_manager.cc:359] Metadata directory not provided
I20260928 02:23:19.146628 27436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:23:19.146955 27436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:23:19.150255 27436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-0/data/instance:
uuid: "717fabead366401fa1a9133f6b911237"
format_stamp: "Formatted at 2026-09-28 02:23:19 on dist-test-slave-h37r"
server_key: "eb8c3f33bebf45a1438ba9e60b6ce25e"
server_key_iv: "df12ebed99afb860cf429cc0b33fc61a"
server_key_version: "encryptionkey@0"
I20260928 02:23:19.151098 27436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-0/wal/instance:
uuid: "717fabead366401fa1a9133f6b911237"
format_stamp: "Formatted at 2026-09-28 02:23:19 on dist-test-slave-h37r"
server_key: "eb8c3f33bebf45a1438ba9e60b6ce25e"
server_key_iv: "df12ebed99afb860cf429cc0b33fc61a"
server_key_version: "encryptionkey@0"
I20260928 02:23:19.156783 27436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:23:19.160420 27451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:23:19.161865 27436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 02:23:19.162065 27436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-0/wal
uuid: "717fabead366401fa1a9133f6b911237"
format_stamp: "Formatted at 2026-09-28 02:23:19 on dist-test-slave-h37r"
server_key: "eb8c3f33bebf45a1438ba9e60b6ce25e"
server_key_iv: "df12ebed99afb860cf429cc0b33fc61a"
server_key_version: "encryptionkey@0"
I20260928 02:23:19.162299 27436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:23:19.174598 27436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:23:19.179293 27436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:23:19.179592 27436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:23:19.198956 27436 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.254:42211
I20260928 02:23:19.198975 27502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.254:42211 every 8 connection(s)
I20260928 02:23:19.200531 27436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-0/data/info.pb
I20260928 02:23:19.204439 27503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:23:19.206321 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 27436
I20260928 02:23:19.206511 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-0/wal/instance
I20260928 02:23:19.206959 4659 external_mini_cluster.cc:1428] Setting key c1a6151994956f8b69a183cc2146c874
I20260928 02:23:19.211369 27503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:23:19 dist-test-slave-h37r krb5kdc[27423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562199, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.140.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.253@KRBTEST.COM" created.
I20260928 02:23:19.234083 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.214658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:42211 (local address 127.0.0.1:53946)
0928 02:23:19.215579 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.215631 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:23:19.217672 (+ 2041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:23:19.218515 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:23:19.218522 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:23:19.219234 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:23:19.219571 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:19.219589 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:19.221252 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:19.221257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:19.222529 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:19.222542 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:19.223465 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:19.223472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:19.223655 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:19.224359 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:23:19.224386 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:23:19.226664 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:23:19.230008 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:19.230019 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:19.230033 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:19.230430 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:19.230908 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:19.230913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:19.230915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:19.231070 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:19.231939 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:23:19.231948 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:23:19.232163 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 02:23:19.232523 (+ 360us) client_negotiation.cc:240] Negotiation successful
0928 02:23:19.233009 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":410,"threads_started":1}
I20260928 02:23:19.234131 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.214657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.4.140.254:42211)
0928 02:23:19.216423 (+ 1766us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:19.216435 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:19.216488 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:23:19.218056 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:19.218062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:19.218294 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:19.218512 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:19.219802 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:19.221094 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:19.222670 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:19.223227 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:19.223429 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:19.226892 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:19.226900 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:19.226913 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:19.226945 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:19.229877 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:19.230561 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:19.230569 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:19.230572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:19.230765 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:19.231182 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:19.231185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:19.231187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:19.231742 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:19.231986 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:19.232665 (+ 679us) server_negotiation.cc:299] Negotiation successful
0928 02:23:19.233189 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":1541,"thread_start_us":1318,"threads_started":1}
I20260928 02:23:19.237269 27503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:23:19.238364 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.237947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38709 (local address 127.0.0.1:39078)
0928 02:23:19.238134 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.238253 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 02:23:19.238624 27455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38709: connect: Connection refused (error 111)
I20260928 02:23:19.240178 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.239794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38709 (local address 127.0.0.1:39094)
0928 02:23:19.239988 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.240062 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 02:23:19.240374 27503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.253:38709: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38709: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.253.keytab.
Entry for principal kudu/127.4.140.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.253.keytab.
I20260928 02:23:19.267611 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.253:38709
--webserver_interface=127.4.140.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.253.keytab
--principal=kudu/127.4.140.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:42365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:42211,127.4.140.253:38709,127.4.140.252:42101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:23:19.289727 27503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } attempt: 1
I20260928 02:23:19.291141 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.290533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38709 (local address 127.0.0.1:39100)
0928 02:23:19.290785 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.290932 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 02:23:19.292286 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.291806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38709 (local address 127.0.0.1:39102)
0928 02:23:19.292007 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.292105 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 02:23:19.292510 27503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.253:38709: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38709: connect: Connection refused (error 111)
I20260928 02:23:19.360816 27503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } attempt: 2
I20260928 02:23:19.362124 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.361545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38709 (local address 127.0.0.1:39108)
0928 02:23:19.361836 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.361953 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 02:23:19.363255 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.362779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38709 (local address 127.0.0.1:39114)
0928 02:23:19.362957 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.363069 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 02:23:19.363480 27503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.253:38709: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38709: connect: Connection refused (error 111)
I20260928 02:23:19.454773 27503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } attempt: 3
I20260928 02:23:19.456110 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.455564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38709 (local address 127.0.0.1:39124)
0928 02:23:19.455801 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.455947 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 02:23:19.457171 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.456778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38709 (local address 127.0.0.1:39136)
0928 02:23:19.456925 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.457005 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 02:23:19.457379 27503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.253:38709: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38709: connect: Connection refused (error 111)
I20260928 02:23:19.600636 27503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } attempt: 4
I20260928 02:23:19.601967 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.601349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38709 (local address 127.0.0.1:39150)
0928 02:23:19.601681 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.601800 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
W20260928 02:23:19.602499 27510 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:23:19.602893 27510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:23:19.603012 27510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260928 02:23:19.603125 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.602697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38709 (local address 127.0.0.1:39152)
0928 02:23:19.602863 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.602968 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 02:23:19.603142 27510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:23:19.603401 27503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.253:38709: Network error: Client connection negotiation failed: client connection to 127.4.140.253:38709: connect: Connection refused (error 111)
W20260928 02:23:19.610666 27510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:23:19.610752 27510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:23:19.610816 27510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:23:19.610863 27510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:23:19.610926 27510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:23:19.620925 27510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:42365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:42211,127.4.140.253:38709,127.4.140.252:42101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.253.keytab
--principal=kudu/127.4.140.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.253:38709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.140.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:23:19.622225 27510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:23:19.623600 27510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:23:19.635403 27518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:19.635459 27516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:19.635360 27515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:23:19.636770 27510 server_base.cc:1034] running on GCE node
I20260928 02:23:19.637496 27510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:23:19.638873 27510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:23:19.640079 27510 hybrid_clock.cc:648] HybridClock initialized: now 1790562199640018 us; error 32 us; skew 500 ppm
I20260928 02:23:19.640372 27510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:23:19 dist-test-slave-h37r krb5kdc[27423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562199, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:23:19.643494 27510 init.cc:377] Logged in from keytab as kudu/127.4.140.253@KRBTEST.COM (short username kudu)
I20260928 02:23:19.645061 27510 webserver.cc:460] Webserver started at http://127.4.140.253:33529/ using document root <none> and password file <none>
I20260928 02:23:19.645759 27510 fs_manager.cc:359] Metadata directory not provided
I20260928 02:23:19.645875 27510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:23:19.646212 27510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:23:19.649220 27510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-1/data/instance:
uuid: "28c116a570af43979288c9850bab5949"
format_stamp: "Formatted at 2026-09-28 02:23:19 on dist-test-slave-h37r"
server_key: "0a0fa4431cd27f5a37a99bb075602d9d"
server_key_iv: "91f59c7a6f398d8f7c2c6f6ca4017996"
server_key_version: "encryptionkey@0"
I20260928 02:23:19.650029 27510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-1/wal/instance:
uuid: "28c116a570af43979288c9850bab5949"
format_stamp: "Formatted at 2026-09-28 02:23:19 on dist-test-slave-h37r"
server_key: "0a0fa4431cd27f5a37a99bb075602d9d"
server_key_iv: "91f59c7a6f398d8f7c2c6f6ca4017996"
server_key_version: "encryptionkey@0"
I20260928 02:23:19.655287 27510 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:23:19.658807 27525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:23:19.660342 27510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:23:19.660529 27510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-1/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-1/wal
uuid: "28c116a570af43979288c9850bab5949"
format_stamp: "Formatted at 2026-09-28 02:23:19 on dist-test-slave-h37r"
server_key: "0a0fa4431cd27f5a37a99bb075602d9d"
server_key_iv: "91f59c7a6f398d8f7c2c6f6ca4017996"
server_key_version: "encryptionkey@0"
I20260928 02:23:19.660750 27510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:23:19.676613 27510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:23:19.680887 27510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:23:19.681174 27510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:23:19.699832 27510 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.253:38709
I20260928 02:23:19.699849 27576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.253:38709 every 8 connection(s)
I20260928 02:23:19.701272 27510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-1/data/info.pb
I20260928 02:23:19.703655 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 27510
I20260928 02:23:19.703842 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-1/wal/instance
I20260928 02:23:19.704296 4659 external_mini_cluster.cc:1428] Setting key 20258e6936f855701d83b19a5f4a07b7
I20260928 02:23:19.705359 27577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:23:19.711993 27577 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:23:19 dist-test-slave-h37r krb5kdc[27423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562199, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.253@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.140.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.140.252@KRBTEST.COM" created.
I20260928 02:23:19.735296 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.715132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53958 (local address 127.4.140.254:42211)
0928 02:23:19.715371 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:19.715378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:19.716516 (+ 1138us) server_negotiation.cc:407] Connection header received
0928 02:23:19.718466 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:19.718476 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:19.718573 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:19.718699 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:19.720445 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:19.721602 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:19.723329 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:19.724145 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:19.724313 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:19.728336 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:19.728344 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:19.728349 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:19.728395 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:19.731664 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:19.732796 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:19.732801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:19.732804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:19.732935 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:19.733472 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:19.733476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:19.733478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:19.733767 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:19.733881 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:19.734878 (+ 997us) server_negotiation.cc:299] Negotiation successful
0928 02:23:19.735064 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 02:23:19.737010 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.715518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:42211 (local address 127.0.0.1:53958)
0928 02:23:19.716337 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.716388 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:23:19.718157 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:23:19.718893 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:23:19.718901 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:23:19.719463 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:23:19.720233 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:19.720254 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:19.721774 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:19.721778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:19.723152 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:19.723164 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:19.724411 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:19.724418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:19.724692 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:19.725455 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:23:19.725479 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:23:19.728033 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:23:19.731832 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:19.731843 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:19.731861 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:19.732619 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:19.733059 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:19.733068 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:19.733074 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:19.733349 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:19.733905 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:23:19.733917 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:23:19.734456 (+ 539us) client_negotiation.cc:769] Sending connection context
0928 02:23:19.734770 (+ 314us) client_negotiation.cc:240] Negotiation successful
0928 02:23:19.735399 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":363,"threads_started":1}
I20260928 02:23:19.738968 27577 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:23:19 dist-test-slave-h37r krb5kdc[27423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562199, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.253@KRBTEST.COM for kudu/127.4.140.253@KRBTEST.COM
I20260928 02:23:19.756711 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.739969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38709 (local address 127.0.0.1:39156)
0928 02:23:19.740197 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.740219 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:23:19.740341 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:23:19.741486 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:23:19.741489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:23:19.741863 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:23:19.741996 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:19.742004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:19.743678 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:19.743687 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:19.745271 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:19.745289 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:19.746157 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:19.746165 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:19.746388 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:19.747050 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:23:19.747078 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:23:19.749763 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:23:19.754180 (+ 4417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:19.754184 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:19.754187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:19.754587 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:19.755029 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:19.755032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:19.755035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:19.755170 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:19.755988 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:23:19.755992 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:23:19.756116 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 02:23:19.756221 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:23:19.756452 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 02:23:19.757246 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.740180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39156 (local address 127.4.140.253:38709)
0928 02:23:19.740951 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:19.740957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:19.740988 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:23:19.741053 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:19.741060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:19.741310 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:19.741497 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:19.742174 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:19.743525 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:19.745440 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:19.745999 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:19.746097 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:19.749997 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:19.750014 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:19.750036 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:19.750078 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:19.754031 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:19.754743 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:19.754755 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:19.754762 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:19.754928 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:19.755299 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:19.755314 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:19.755320 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:19.755813 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:19.756065 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:19.756469 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 02:23:19.756757 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":439,"threads_started":1}
I20260928 02:23:19.759995 27577 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:23:19.761204 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.760700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57290)
0928 02:23:19.760895 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.761053 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 02:23:19.761442 27526 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
I20260928 02:23:19.762897 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.762532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57304)
0928 02:23:19.762723 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.762798 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260928 02:23:19.763114 27577 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:42101: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.140.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.252.keytab.
Entry for principal kudu/127.4.140.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.252.keytab.
I20260928 02:23:19.771593 4659 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
/tmp/dist-test-taskx4rqyh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.252:42101
--webserver_interface=127.4.140.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.252.keytab
--principal=kudu/127.4.140.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.140.212:42365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.140.254:42211,127.4.140.253:38709,127.4.140.252:42101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:23:19.812422 27577 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } attempt: 1
I20260928 02:23:19.813644 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.813177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57312)
0928 02:23:19.813381 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.813490 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 02:23:19.814736 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.814414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57324)
0928 02:23:19.814561 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.814634 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 02:23:19.814976 27577 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:42101: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
I20260928 02:23:19.883220 27577 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } attempt: 2
I20260928 02:23:19.884706 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.884187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57340)
0928 02:23:19.884440 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.884538 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 02:23:19.885812 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.885460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57344)
0928 02:23:19.885613 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.885693 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 02:23:19.886008 27577 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:42101: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
I20260928 02:23:19.902650 27503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } attempt: 5
Sep 28 02:23:19 dist-test-slave-h37r krb5kdc[27423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562199, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for kudu/127.4.140.253@KRBTEST.COM
I20260928 02:23:19.917506 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.903400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38709 (local address 127.0.0.1:39168)
0928 02:23:19.903600 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.903625 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:23:19.903740 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:23:19.904161 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:23:19.904164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:23:19.904582 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:23:19.904763 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:19.904772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:19.906242 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:19.906246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:19.907101 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:19.907114 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:19.908317 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:19.908321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:19.908501 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:19.909166 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:23:19.909184 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:23:19.911241 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:23:19.915155 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:19.915159 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:19.915162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:19.915554 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:19.915991 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:19.915995 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:19.915997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:19.916128 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:19.916760 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:23:19.916764 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:23:19.916916 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 02:23:19.917026 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:23:19.917234 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 02:23:19.917743 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.903504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39168 (local address 127.4.140.253:38709)
0928 02:23:19.903704 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:19.903711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:19.903737 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:23:19.903865 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:19.903871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:19.904005 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:19.904128 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:19.904936 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:19.906096 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:19.907269 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:19.908151 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:19.908331 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:19.911468 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:19.911474 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:19.911479 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:19.911523 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:19.914991 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:19.915680 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:19.915686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:19.915689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:19.915839 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:19.916236 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:19.916241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:19.916244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:19.916619 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:19.916782 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:19.917186 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 02:23:19.917436 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 02:23:19.918761 27503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:23:19.919767 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.919402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57350)
0928 02:23:19.919558 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.919633 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 02:23:19.920758 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.920454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57358)
0928 02:23:19.920583 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.920654 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 02:23:19.920964 27503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:42101: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
I20260928 02:23:19.972270 27503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } attempt: 1
I20260928 02:23:19.973476 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.972996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57368)
0928 02:23:19.973202 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.973304 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 02:23:19.974417 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.974103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57380)
0928 02:23:19.974243 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.974314 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 02:23:19.974618 27503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:42101: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
I20260928 02:23:19.977195 27577 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } attempt: 3
I20260928 02:23:19.978240 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.977736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57392)
0928 02:23:19.977962 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.978067 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 02:23:19.979380 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:19.978889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57408)
0928 02:23:19.979122 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:19.979213 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260928 02:23:19.979584 27577 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:42101: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
I20260928 02:23:20.042834 27503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } attempt: 2
I20260928 02:23:20.044095 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.043543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57416)
0928 02:23:20.043751 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:20.043865 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 02:23:20.045130 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.044784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57430)
0928 02:23:20.044935 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:20.045022 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:23:20.045336 27503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:42101: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
W20260928 02:23:20.116407 27584 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:23:20.116801 27584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:23:20.116925 27584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:23:20.117061 27584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 02:23:20.122807 27577 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } attempt: 4
I20260928 02:23:20.124083 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.123559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57438)
0928 02:23:20.123781 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:20.123877 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 02:23:20.125089 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.124737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57454)
0928 02:23:20.124905 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:20.124982 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 02:23:20.125280 27584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:23:20.125293 27577 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:42101: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
W20260928 02:23:20.125384 27584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:23:20.125473 27584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:23:20.125569 27584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:23:20.125661 27584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:23:20.136018 27584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.140.212:42365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.140.254:42211,127.4.140.253:38709,127.4.140.252:42101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/krb5kdc/kudu_127.4.140.252.keytab
--principal=kudu/127.4.140.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.140.252:42101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.140.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:23:20.137346 27584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:23:20.138756 27584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:23:20.150897 27589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:20.150862 27590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:23:20.150887 27592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:23:20.151551 27503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } attempt: 3
I20260928 02:23:20.152320 27584 server_base.cc:1034] running on GCE node
I20260928 02:23:20.152797 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.152164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57464)
0928 02:23:20.152511 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:20.152606 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
I20260928 02:23:20.153287 27584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:23:20.153934 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.153567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57470)
0928 02:23:20.153753 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:20.153822 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260928 02:23:20.154222 27503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.140.252:42101: Network error: Client connection negotiation failed: client connection to 127.4.140.252:42101: connect: Connection refused (error 111)
I20260928 02:23:20.154944 27584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:23:20.156152 27584 hybrid_clock.cc:648] HybridClock initialized: now 1790562200156050 us; error 78 us; skew 500 ppm
I20260928 02:23:20.156512 27584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:23:20 dist-test-slave-h37r krb5kdc[27423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562200, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:23:20.159721 27584 init.cc:377] Logged in from keytab as kudu/127.4.140.252@KRBTEST.COM (short username kudu)
I20260928 02:23:20.161223 27584 webserver.cc:460] Webserver started at http://127.4.140.252:43103/ using document root <none> and password file <none>
I20260928 02:23:20.161924 27584 fs_manager.cc:359] Metadata directory not provided
I20260928 02:23:20.162060 27584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:23:20.162387 27584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:23:20.165688 27584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-2/data/instance:
uuid: "8e53da5be01d40cc8be3f23f3a580b71"
format_stamp: "Formatted at 2026-09-28 02:23:20 on dist-test-slave-h37r"
server_key: "bc327c040146d33e71b163fd40a9c1f7"
server_key_iv: "0b9c00c651ba337015848ee6c69a74e2"
server_key_version: "encryptionkey@0"
I20260928 02:23:20.166592 27584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-2/wal/instance:
uuid: "8e53da5be01d40cc8be3f23f3a580b71"
format_stamp: "Formatted at 2026-09-28 02:23:20 on dist-test-slave-h37r"
server_key: "bc327c040146d33e71b163fd40a9c1f7"
server_key_iv: "0b9c00c651ba337015848ee6c69a74e2"
server_key_version: "encryptionkey@0"
I20260928 02:23:20.171998 27584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:23:20.175496 27599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:23:20.176944 27584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 02:23:20.177139 27584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-2/data,/tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-2/wal
uuid: "8e53da5be01d40cc8be3f23f3a580b71"
format_stamp: "Formatted at 2026-09-28 02:23:20 on dist-test-slave-h37r"
server_key: "bc327c040146d33e71b163fd40a9c1f7"
server_key_iv: "0b9c00c651ba337015848ee6c69a74e2"
server_key_version: "encryptionkey@0"
I20260928 02:23:20.177364 27584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:23:20.192014 27584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:23:20.196381 27584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:23:20.196651 27584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:23:20.216078 27584 rpc_server.cc:307] RPC server started. Bound to: 127.4.140.252:42101
I20260928 02:23:20.216099 27650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.140.252:42101 every 8 connection(s)
I20260928 02:23:20.217468 27584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-2/data/info.pb
I20260928 02:23:20.221486 27651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:23:20.227662 4659 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu as pid 27584
I20260928 02:23:20.227871 4659 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4rqyh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561835360118-4659-0/minicluster-data/master-2/wal/instance
I20260928 02:23:20.228310 4659 external_mini_cluster.cc:1428] Setting key 9618562e2b6cf9145b9b49d76a83ebdd
I20260928 02:23:20.228243 27651 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:23:20 dist-test-slave-h37r krb5kdc[27423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562200, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.252@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
Sep 28 02:23:20 dist-test-slave-h37r krb5kdc[27423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.252@KRBTEST.COM
I20260928 02:23:20.249862 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.231123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.4.140.254:42211)
0928 02:23:20.231341 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:20.231348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:20.232105 (+ 757us) server_negotiation.cc:407] Connection header received
0928 02:23:20.233342 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:20.233350 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:20.233454 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:20.233574 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:20.235470 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.236618 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.238722 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.239400 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.239535 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:20.243657 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:20.243666 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:20.243675 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:20.243721 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:20.246524 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.247371 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.247375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.247377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.247494 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.248001 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.248004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.248006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.248400 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:20.248568 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:20.249387 (+ 819us) server_negotiation.cc:299] Negotiation successful
0928 02:23:20.249584 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:23:20.251142 27655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.231122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.254:42211 (local address 127.0.0.1:53962)
0928 02:23:20.231949 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:20.231999 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:23:20.233005 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:23:20.233818 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:23:20.233827 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:23:20.234720 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:23:20.235293 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:20.235317 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.236776 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.236780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:20.238576 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:20.238592 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.239622 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.239627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:20.239888 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:20.240732 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:23:20.240764 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:23:20.243451 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:23:20.246652 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:20.246662 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:20.246675 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:20.247250 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:20.247607 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:20.247615 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:20.247619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:20.247857 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:20.248529 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:23:20.248535 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:23:20.248936 (+ 401us) client_negotiation.cc:769] Sending connection context
0928 02:23:20.249221 (+ 285us) client_negotiation.cc:240] Negotiation successful
0928 02:23:20.249769 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":57,"thread_start_us":404,"threads_started":1}
I20260928 02:23:20.251490 27654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.230156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.4.140.252:42101)
0928 02:23:20.230977 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:20.230990 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:20.231038 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:23:20.232995 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:20.233025 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:20.233358 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:20.233595 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:20.235482 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.237068 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.238830 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.239698 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.239868 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:20.244521 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:20.244534 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:20.244551 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:20.244590 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:20.248038 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.248789 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.248797 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.248799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.248982 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.249395 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.249399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.249402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.249827 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:20.250135 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:20.250866 (+ 731us) server_negotiation.cc:299] Negotiation successful
0928 02:23:20.251039 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":579,"spinlock_wait_cycles":43264,"thread_start_us":359,"threads_started":1}
I20260928 02:23:20.252909 27651 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:23:20 dist-test-slave-h37r krb5kdc[27423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562200, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.252@KRBTEST.COM for kudu/127.4.140.253@KRBTEST.COM
Sep 28 02:23:20 dist-test-slave-h37r krb5kdc[27423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.254@KRBTEST.COM
I20260928 02:23:20.266511 27655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.253777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.253:38709 (local address 127.0.0.1:39176)
0928 02:23:20.253982 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:20.254002 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:23:20.254110 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:23:20.254499 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:23:20.254505 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:23:20.254941 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:23:20.255090 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:20.255098 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.256116 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.256123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:20.257014 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:20.257026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.257728 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.257734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:20.257942 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:20.258517 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:23:20.258533 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:23:20.260748 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:23:20.263982 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:20.263989 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:20.263993 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:20.264411 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:20.264779 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:20.264785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:20.264788 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:20.264943 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:20.265536 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:23:20.265541 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:23:20.265700 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 02:23:20.265851 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 02:23:20.266137 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 02:23:20.266538 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.253771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39176 (local address 127.4.140.253:38709)
0928 02:23:20.253983 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:20.253993 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:20.254062 (+ 69us) server_negotiation.cc:407] Connection header received
0928 02:23:20.254230 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:20.254234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:20.254319 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:20.254440 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:20.255235 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.255989 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.257163 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.257587 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.257721 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:20.260944 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:20.260949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:20.260952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:20.260985 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:20.263819 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.264530 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.264535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.264539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.264668 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.265054 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.265060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.265063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.265410 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:20.265571 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:20.265968 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 02:23:20.266172 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:23:20.267704 27651 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:23:20.268743 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.254716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53976 (local address 127.4.140.254:42211)
0928 02:23:20.254931 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:20.254938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:20.254953 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:23:20.255024 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:20.255027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:20.255078 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:20.255177 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:20.256376 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.257113 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.258294 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.258716 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.258856 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:20.262909 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:20.262914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:20.262917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:20.262949 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:20.265809 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.266436 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.266443 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.266446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.266697 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.267164 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.267169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.267172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.267512 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:20.267761 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:20.268172 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 02:23:20.268427 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 02:23:20 dist-test-slave-h37r krb5kdc[27423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562200, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.252@KRBTEST.COM for kudu/127.4.140.252@KRBTEST.COM
Sep 28 02:23:20 dist-test-slave-h37r krb5kdc[27423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.140.253@KRBTEST.COM
I20260928 02:23:20.281368 27654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.268929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57494 (local address 127.4.140.252:42101)
0928 02:23:20.269077 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:20.269082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:20.269106 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:23:20.269143 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:20.269146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:20.269217 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:20.269319 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:20.270055 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.271059 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.272288 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.272703 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.272817 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:20.276144 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:20.276150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:20.276154 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:20.276190 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:20.278747 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.279381 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.279389 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.279392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.279516 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.279870 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.279874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.279878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.280215 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:20.280366 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:20.280730 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 02:23:20.281009 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:23:20.281494 27655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.268513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57494)
0928 02:23:20.268827 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:20.268850 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:23:20.268954 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:23:20.269369 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:23:20.269372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:23:20.269766 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:23:20.269930 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:20.269939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.271172 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.271179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:20.272146 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:20.272160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.272849 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.272854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:20.273086 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:20.273641 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:23:20.273657 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:23:20.275983 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:23:20.278892 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:20.278898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:20.278902 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:20.279239 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:20.279619 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:20.279625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:20.279628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:20.279761 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:20.280348 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:23:20.280353 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:23:20.280492 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 02:23:20.280633 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 02:23:20.281132 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":177}
I20260928 02:23:20.284983 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.270394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39188 (local address 127.4.140.253:38709)
0928 02:23:20.270558 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:20.270563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:20.270623 (+ 60us) server_negotiation.cc:407] Connection header received
0928 02:23:20.270749 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:20.270754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:20.270818 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:20.270903 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:20.272024 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.272891 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.274038 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.274683 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.274823 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:20.278841 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:20.278847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:20.278852 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:20.278893 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:20.281835 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.282856 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.282861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.282864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.283026 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.283396 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.283400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.283403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.283763 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:20.284081 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:20.284255 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 02:23:20.284480 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:23:20.287847 27651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71: Bootstrap starting.
I20260928 02:23:20.289847 4659 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:23:20.291464 27651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71: Neither blocks nor log segments found. Creating new log.
I20260928 02:23:20.292658 27651 log.cc:824] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71: Log is configured to *not* fsync() on all Append() calls
I20260928 02:23:20.292145 27478 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "717fabead366401fa1a9133f6b911237"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53976
I20260928 02:23:20.295392 27651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71: No bootstrap required, opened a new log
I20260928 02:23:20.302438 27651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717fabead366401fa1a9133f6b911237" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } } peers { permanent_uuid: "28c116a570af43979288c9850bab5949" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } } peers { permanent_uuid: "8e53da5be01d40cc8be3f23f3a580b71" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } }
I20260928 02:23:20.302778 27651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e53da5be01d40cc8be3f23f3a580b71, State: Initialized, Role: FOLLOWER
I20260928 02:23:20.303248 27651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717fabead366401fa1a9133f6b911237" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } } peers { permanent_uuid: "28c116a570af43979288c9850bab5949" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } } peers { permanent_uuid: "8e53da5be01d40cc8be3f23f3a580b71" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } }
I20260928 02:23:20.304271 27657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717fabead366401fa1a9133f6b911237" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } } peers { permanent_uuid: "28c116a570af43979288c9850bab5949" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } } peers { permanent_uuid: "8e53da5be01d40cc8be3f23f3a580b71" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } } }
I20260928 02:23:20.304673 27657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:23:20.305265 27651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:23:20.314359 27668 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:23:20.314492 27668 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:23:20.312789 27584 master_runner.cc:186] Error getting master registration for 127.4.140.254:42211: OK, instance_id { permanent_uuid: "717fabead366401fa1a9133f6b911237" instance_seqno: 1790562199193333 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:23:20.324008 27584 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:23:20.331403 27503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } attempt: 4
Sep 28 02:23:20 dist-test-slave-h37r krb5kdc[27423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562199, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.254@KRBTEST.COM for kudu/127.4.140.252@KRBTEST.COM
I20260928 02:23:20.348644 27654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.332279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57496 (local address 127.4.140.252:42101)
0928 02:23:20.332495 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:20.332501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:20.332522 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:23:20.332687 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:20.332693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:20.332769 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:20.332876 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:20.333839 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.335086 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.336987 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.337738 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.337866 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:20.341954 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:20.341960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:20.341965 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:20.342009 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:20.345581 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.346381 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.346387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.346391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.346531 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.346973 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.346978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.346981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.347436 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:20.347578 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:20.348015 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 02:23:20.348260 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:23:20.348645 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.332201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57496)
0928 02:23:20.332423 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:20.332445 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:23:20.332573 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:23:20.332910 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:23:20.332915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:23:20.333440 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:23:20.333658 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:20.333672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.335274 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.335279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:20.336818 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:20.336832 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.337910 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.337915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:20.338229 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:20.338967 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:23:20.338994 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:23:20.341759 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:23:20.345722 (+ 3963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:20.345728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:20.345732 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:20.346261 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:20.346656 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:20.346662 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:20.346666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:20.346860 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:20.347559 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:23:20.347564 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:23:20.347739 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 02:23:20.347937 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 02:23:20.348237 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 02:23:20.354884 27503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237: Bootstrap starting.
I20260928 02:23:20.359182 27503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237: Neither blocks nor log segments found. Creating new log.
I20260928 02:23:20.360564 27503 log.cc:824] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237: Log is configured to *not* fsync() on all Append() calls
I20260928 02:23:20.364127 27503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237: No bootstrap required, opened a new log
I20260928 02:23:20.371944 27503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717fabead366401fa1a9133f6b911237" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } } peers { permanent_uuid: "28c116a570af43979288c9850bab5949" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } } peers { permanent_uuid: "8e53da5be01d40cc8be3f23f3a580b71" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } }
I20260928 02:23:20.372462 27503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717fabead366401fa1a9133f6b911237, State: Initialized, Role: FOLLOWER
I20260928 02:23:20.373034 27503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717fabead366401fa1a9133f6b911237" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } } peers { permanent_uuid: "28c116a570af43979288c9850bab5949" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } } peers { permanent_uuid: "8e53da5be01d40cc8be3f23f3a580b71" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } }
I20260928 02:23:20.374074 27671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717fabead366401fa1a9133f6b911237" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } } peers { permanent_uuid: "28c116a570af43979288c9850bab5949" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } } peers { permanent_uuid: "8e53da5be01d40cc8be3f23f3a580b71" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } } }
I20260928 02:23:20.374472 27671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:23:20.375507 27503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:23:20.375643 27478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 02:23:20.375864 27478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:23:20.378479 27478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717fabead366401fa1a9133f6b911237" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } } peers { permanent_uuid: "28c116a570af43979288c9850bab5949" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } } peers { permanent_uuid: "8e53da5be01d40cc8be3f23f3a580b71" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } }
I20260928 02:23:20.381263 27478 leader_election.cc:288] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [CANDIDATE]: Term 1 election: Requested vote from peers 28c116a570af43979288c9850bab5949 (127.4.140.253:38709), 8e53da5be01d40cc8be3f23f3a580b71 (127.4.140.252:42101)
I20260928 02:23:20.381392 27626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "717fabead366401fa1a9133f6b911237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8e53da5be01d40cc8be3f23f3a580b71"
I20260928 02:23:20.381917 27626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:23:20.386437 27626 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717fabead366401fa1a9133f6b911237 in term 1.
I20260928 02:23:20.387223 27452 leader_election.cc:302] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717fabead366401fa1a9133f6b911237, 8e53da5be01d40cc8be3f23f3a580b71; no voters:
I20260928 02:23:20.387806 27671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:23:20.396255 27671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [term 1 LEADER]: Becoming Leader. State: Replica: 717fabead366401fa1a9133f6b911237, State: Running, Role: LEADER
I20260928 02:23:20.397076 27671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717fabead366401fa1a9133f6b911237" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } } peers { permanent_uuid: "28c116a570af43979288c9850bab5949" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } } peers { permanent_uuid: "8e53da5be01d40cc8be3f23f3a580b71" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } }
I20260928 02:23:20.412071 27680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 717fabead366401fa1a9133f6b911237. Latest consensus state: current_term: 1 leader_uuid: "717fabead366401fa1a9133f6b911237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717fabead366401fa1a9133f6b911237" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } } peers { permanent_uuid: "28c116a570af43979288c9850bab5949" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } } peers { permanent_uuid: "8e53da5be01d40cc8be3f23f3a580b71" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } } }
I20260928 02:23:20.412674 27680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [sys.catalog]: This master's current role is: LEADER
I20260928 02:23:20.424535 27577 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } attempt: 5
I20260928 02:23:20.432281 27688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260928 02:23:20.433233 27457 tablet.cc:2369] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:23:20 dist-test-slave-h37r krb5kdc[27423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562199, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.140.253@KRBTEST.COM for kudu/127.4.140.252@KRBTEST.COM
W20260928 02:23:20.449062 27605 tablet.cc:2369] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:23:20.457839 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.407070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.4.140.254:42211)
0928 02:23:20.407268 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:20.407274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:20.417255 (+ 9981us) server_negotiation.cc:407] Connection header received
0928 02:23:20.417462 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:20.417472 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:20.417596 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:20.417735 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:20.423499 (+ 5764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.425056 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.436179 (+ 11123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.437254 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.437405 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:20.447484 (+ 10079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:20.447494 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:20.447502 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:20.447563 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:20.451466 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.454202 (+ 2736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.454208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.454212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.454400 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.455743 (+ 1343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.455749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.455753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.456199 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:20.456481 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:20.457147 (+ 666us) server_negotiation.cc:299] Negotiation successful
0928 02:23:20.457414 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:23:20.458003 27688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:23:20.458410 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.413995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39194 (local address 127.4.140.253:38709)
0928 02:23:20.414232 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:20.414242 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:20.422636 (+ 8394us) server_negotiation.cc:407] Connection header received
0928 02:23:20.423079 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:20.423087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:20.423171 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:20.423245 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:20.427649 (+ 4404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.428905 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.432107 (+ 3202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.432776 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.432869 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:20.442642 (+ 9773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:20.442653 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:20.442661 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:20.442722 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:20.452462 (+ 9740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.454063 (+ 1601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.454070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.454073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.454326 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.455081 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.455086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.455089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.455667 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:20.456075 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:20.456998 (+ 923us) server_negotiation.cc:299] Negotiation successful
0928 02:23:20.457388 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:23:20.466656 27654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.418207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57504 (local address 127.4.140.252:42101)
0928 02:23:20.418442 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:20.418451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:20.427867 (+ 9416us) server_negotiation.cc:407] Connection header received
0928 02:23:20.428753 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:20.428759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:20.428858 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:20.428931 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:20.438331 (+ 9400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.439419 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.452012 (+ 12593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.452736 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.452838 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:20.456184 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:20.456190 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:20.456194 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:20.456235 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:20.459866 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.462895 (+ 3029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.462902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.462905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.463075 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.463575 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.463581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.463585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.464029 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:20.464277 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:20.465996 (+ 1719us) server_negotiation.cc:299] Negotiation successful
0928 02:23:20.466250 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:23:20.471113 27626 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71 [term 1 FOLLOWER]: Refusing update from remote peer 717fabead366401fa1a9133f6b911237: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:23:20.471202 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.425223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.140.252:42101 (local address 127.0.0.1:57506)
0928 02:23:20.425682 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 02:23:20.425706 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:23:20.425824 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:23:20.435025 (+ 9201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:23:20.435031 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:23:20.435516 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:23:20.435722 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:20.435734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.440168 (+ 4434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.440175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:20.441636 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:23:20.441653 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.442636 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.442644 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:23:20.442933 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:20.443637 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:23:20.443660 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:23:20.446670 (+ 3010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:23:20.464681 (+ 18011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:20.464687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:20.464691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:20.465238 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:20.469162 (+ 3924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:23:20.469167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:23:20.469171 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:23:20.469356 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:23:20.470154 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:23:20.470159 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:23:20.470346 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 02:23:20.470484 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 02:23:20.470807 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289}
I20260928 02:23:20.474046 27691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.425671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57506 (local address 127.4.140.252:42101)
0928 02:23:20.431734 (+ 6063us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:20.431742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:20.431771 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:23:20.431825 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:20.431831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:20.431925 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:20.435882 (+ 3957us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:20.436206 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.437477 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.441832 (+ 4355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.442441 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.443277 (+ 836us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:20.452736 (+ 9459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:23:20.452743 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:23:20.452747 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:23:20.452792 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:23:20.461299 (+ 8507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.465428 (+ 4129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.465435 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.465439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.465607 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:23:20.469513 (+ 3906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:23:20.469518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:23:20.469521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:23:20.469888 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:23:20.473221 (+ 3333us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:20.473346 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 02:23:20.473593 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5894,"thread_start_us":5707,"threads_started":1}
I20260928 02:23:20.474120 27680 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e53da5be01d40cc8be3f23f3a580b71" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:23:20.484592 27577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c116a570af43979288c9850bab5949: Bootstrap starting.
I20260928 02:23:20.491647 27577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28c116a570af43979288c9850bab5949: Neither blocks nor log segments found. Creating new log.
I20260928 02:23:20.493103 27577 log.cc:824] T 00000000000000000000000000000000 P 28c116a570af43979288c9850bab5949: Log is configured to *not* fsync() on all Append() calls
I20260928 02:23:20.496644 27436 master_runner.cc:186] Error getting master registration for 127.4.140.253:38709: OK, instance_id { permanent_uuid: "28c116a570af43979288c9850bab5949" instance_seqno: 1790562199694518 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:23:20.510494 27436 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:23:20.514912 27577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c116a570af43979288c9850bab5949: No bootstrap required, opened a new log
I20260928 02:23:20.520429 27657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 717fabead366401fa1a9133f6b911237. Latest consensus state: current_term: 1 leader_uuid: "717fabead366401fa1a9133f6b911237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717fabead366401fa1a9133f6b911237" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } } peers { permanent_uuid: "28c116a570af43979288c9850bab5949" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } } peers { permanent_uuid: "8e53da5be01d40cc8be3f23f3a580b71" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } } }
I20260928 02:23:20.520722 27657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:23:20.523121 27695 mvcc.cc:205] Tried to move back new op lower bound from 7334142773113417728 to 7334142772834373632. Current Snapshot: MvccSnapshot[applied={T|T < 7334142773113417728}]
I20260928 02:23:20.532117 27577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28c116a570af43979288c9850bab5949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717fabead366401fa1a9133f6b911237" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } } peers { permanent_uuid: "28c116a570af43979288c9850bab5949" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } } peers { permanent_uuid: "8e53da5be01d40cc8be3f23f3a580b71" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } }
I20260928 02:23:20.532846 27577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28c116a570af43979288c9850bab5949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28c116a570af43979288c9850bab5949, State: Initialized, Role: FOLLOWER
I20260928 02:23:20.533872 27577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28c116a570af43979288c9850bab5949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717fabead366401fa1a9133f6b911237" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } } peers { permanent_uuid: "28c116a570af43979288c9850bab5949" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } } peers { permanent_uuid: "8e53da5be01d40cc8be3f23f3a580b71" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } }
I20260928 02:23:20.545104 27552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "717fabead366401fa1a9133f6b911237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "28c116a570af43979288c9850bab5949"
I20260928 02:23:20.546633 27551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28c116a570af43979288c9850bab5949 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:23:20.550421 27577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28c116a570af43979288c9850bab5949 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:23:20.550868 27671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "717fabead366401fa1a9133f6b911237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717fabead366401fa1a9133f6b911237" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } } peers { permanent_uuid: "28c116a570af43979288c9850bab5949" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } } peers { permanent_uuid: "8e53da5be01d40cc8be3f23f3a580b71" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } } }
I20260928 02:23:20.551245 27671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [sys.catalog]: This master's current role is: LEADER
I20260928 02:23:20.551954 27551 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 28c116a570af43979288c9850bab5949 [term 1 FOLLOWER]: Refusing update from remote peer 717fabead366401fa1a9133f6b911237: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:23:20.555042 27657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "717fabead366401fa1a9133f6b911237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717fabead366401fa1a9133f6b911237" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } } peers { permanent_uuid: "28c116a570af43979288c9850bab5949" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } } peers { permanent_uuid: "8e53da5be01d40cc8be3f23f3a580b71" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } } }
I20260928 02:23:20.555310 27657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:23:20.555747 27552 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 28c116a570af43979288c9850bab5949 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717fabead366401fa1a9133f6b911237 in term 1.
I20260928 02:23:20.557454 27688 catalog_manager.cc:1347] Generated new cluster ID: 48f27ece03914a17bad74e81dc62e044
I20260928 02:23:20.557555 27688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:23:20.558102 27680 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28c116a570af43979288c9850bab5949" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:23:20.578956 27703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c116a570af43979288c9850bab5949 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 717fabead366401fa1a9133f6b911237. Latest consensus state: current_term: 1 leader_uuid: "717fabead366401fa1a9133f6b911237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717fabead366401fa1a9133f6b911237" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } } peers { permanent_uuid: "28c116a570af43979288c9850bab5949" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } } peers { permanent_uuid: "8e53da5be01d40cc8be3f23f3a580b71" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } } }
I20260928 02:23:20.579437 27703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c116a570af43979288c9850bab5949 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:23:20.579972 27709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c116a570af43979288c9850bab5949 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "717fabead366401fa1a9133f6b911237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717fabead366401fa1a9133f6b911237" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } } peers { permanent_uuid: "28c116a570af43979288c9850bab5949" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } } peers { permanent_uuid: "8e53da5be01d40cc8be3f23f3a580b71" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } } }
I20260928 02:23:20.580390 27709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c116a570af43979288c9850bab5949 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:23:20.580657 27701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c116a570af43979288c9850bab5949 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "717fabead366401fa1a9133f6b911237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717fabead366401fa1a9133f6b911237" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } } peers { permanent_uuid: "28c116a570af43979288c9850bab5949" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } } peers { permanent_uuid: "8e53da5be01d40cc8be3f23f3a580b71" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } } }
I20260928 02:23:20.581082 27701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c116a570af43979288c9850bab5949 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:23:20.581809 27688 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:23:20.582154 27671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "717fabead366401fa1a9133f6b911237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717fabead366401fa1a9133f6b911237" member_type: VOTER last_known_addr { host: "127.4.140.254" port: 42211 } } peers { permanent_uuid: "28c116a570af43979288c9850bab5949" member_type: VOTER last_known_addr { host: "127.4.140.253" port: 38709 } } peers { permanent_uuid: "8e53da5be01d40cc8be3f23f3a580b71" member_type: VOTER last_known_addr { host: "127.4.140.252" port: 42101 } } }
I20260928 02:23:20.582466 27671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237 [sys.catalog]: This master's current role is: LEADER
I20260928 02:23:20.583774 27688 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:23:20.606977 27688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 717fabead366401fa1a9133f6b911237: Generated new TSK 0
I20260928 02:23:20.608103 27688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:23:20.667795 27718 catalog_manager.cc:1259] Loaded cluster ID: 48f27ece03914a17bad74e81dc62e044
I20260928 02:23:20.667927 27718 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 28c116a570af43979288c9850bab5949: loading cluster ID for follower catalog manager: success
I20260928 02:23:20.671746 27718 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 28c116a570af43979288c9850bab5949: acquiring CA information for follower catalog manager: success
I20260928 02:23:20.674445 27718 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 28c116a570af43979288c9850bab5949: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 02:23:20.772199 27581 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:20.757032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39210 (local address 127.4.140.253:38709)
0928 02:23:20.757273 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:20.757280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:20.768976 (+ 11696us) server_negotiation.cc:407] Connection header received
0928 02:23:20.770557 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:20.770564 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:20.770694 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:20.770771 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:20.771927 (+ 1156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39210: BlockingRecv error: recv got EOF from 127.0.0.1:39210 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 02:23:20.774933 27507 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:20.750128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.4.140.254:42211)
0928 02:23:20.750382 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:20.750389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:20.768466 (+ 18077us) server_negotiation.cc:407] Connection header received
0928 02:23:20.772713 (+ 4247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:20.772720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:20.772804 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:20.772876 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:20.774682 (+ 1806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53998: BlockingRecv error: recv got EOF from 127.0.0.1:53998 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 02:23:20.778132 27691 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:20.769692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57518 (local address 127.4.140.252:42101)
0928 02:23:20.769922 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:20.769929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:20.775887 (+ 5958us) server_negotiation.cc:407] Connection header received
0928 02:23:20.776792 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:20.776798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:20.776885 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:20.776971 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:20.777889 (+ 918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57518: BlockingRecv error: recv got EOF from 127.0.0.1:57518 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:23:20.840723 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.825908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39212 (local address 127.4.140.253:38709)
0928 02:23:20.826108 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:20.826114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:20.832194 (+ 6080us) server_negotiation.cc:407] Connection header received
0928 02:23:20.832819 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:20.832826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:20.833075 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:20.833235 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:20.833997 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.835235 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.837438 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.838145 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.838248 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:20.839068 (+ 820us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:20.839824 (+ 756us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:20.839967 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:20.840230 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 02:23:20.840338 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 02:23:20.890775 27691 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:20.877469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57532 (local address 127.4.140.252:42101)
0928 02:23:20.877722 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:20.877732 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:20.888569 (+ 10837us) server_negotiation.cc:407] Connection header received
0928 02:23:20.889117 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:20.889124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:20.889253 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:20.889345 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:20.890489 (+ 1144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57532: BlockingRecv error: recv got EOF from 127.0.0.1:57532 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:23:20.943434 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:20.929016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.4.140.253:38709)
0928 02:23:20.929216 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:20.929224 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:20.936166 (+ 6942us) server_negotiation.cc:407] Connection header received
0928 02:23:20.936730 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:20.936739 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:20.936865 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:20.936934 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:20.937633 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.938850 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.940964 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:20.941680 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:20.941793 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:20.942459 (+ 666us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:20.942660 (+ 201us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:20.942731 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:20.942965 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 02:23:20.943064 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 02:23:20.989466 27691 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:20.977727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57548 (local address 127.4.140.252:42101)
0928 02:23:20.977924 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:20.977932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:20.985394 (+ 7462us) server_negotiation.cc:407] Connection header received
0928 02:23:20.985670 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:20.985676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:20.985807 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:20.985895 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:20.989219 (+ 3324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57548: BlockingRecv error: recv got EOF from 127.0.0.1:57548 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:23:21.017951 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:21.010586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39230 (local address 127.4.140.253:38709)
0928 02:23:21.010801 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:21.010808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:21.011397 (+ 589us) server_negotiation.cc:407] Connection header received
0928 02:23:21.011613 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:21.011621 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:21.011730 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:21.011830 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:21.012403 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.013536 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.015605 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.016309 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.016471 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:21.017017 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:21.017169 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:21.017277 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:21.017451 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 02:23:21.017547 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 02:23:21.029685 27691 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:21.027142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57550 (local address 127.4.140.252:42101)
0928 02:23:21.027417 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:21.027425 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:21.027973 (+ 548us) server_negotiation.cc:407] Connection header received
0928 02:23:21.028208 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:21.028217 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:21.028346 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:21.028477 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:21.029418 (+ 941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57550: BlockingRecv error: recv got EOF from 127.0.0.1:57550 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
I20260928 02:23:21.045573 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:21.037935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39236 (local address 127.4.140.253:38709)
0928 02:23:21.038186 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:21.038206 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:21.038802 (+ 596us) server_negotiation.cc:407] Connection header received
0928 02:23:21.039020 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:21.039027 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:21.039154 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:21.039263 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:21.039778 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.041083 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.043150 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.043963 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.044136 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:21.044637 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:21.044788 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:21.044908 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:21.045084 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 02:23:21.045204 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
W20260928 02:23:21.060473 27691 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:21.057859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57558 (local address 127.4.140.252:42101)
0928 02:23:21.058134 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:21.058141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:21.058780 (+ 639us) server_negotiation.cc:407] Connection header received
0928 02:23:21.058997 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:21.059004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:21.059123 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:21.059250 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:21.060204 (+ 954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57558: BlockingRecv error: recv got EOF from 127.0.0.1:57558 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
I20260928 02:23:21.079963 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:21.072469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39244 (local address 127.4.140.253:38709)
0928 02:23:21.072663 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:21.072671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:21.073211 (+ 540us) server_negotiation.cc:407] Connection header received
0928 02:23:21.073441 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:21.073448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:21.073562 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:21.073694 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:21.074157 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.075364 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.077431 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.078210 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.078363 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:21.078865 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:21.079043 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:21.079207 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:21.079398 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 02:23:21.079500 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 02:23:21.093303 27691 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:21.088070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.4.140.252:42101)
0928 02:23:21.088317 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:21.088323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:21.091588 (+ 3265us) server_negotiation.cc:407] Connection header received
0928 02:23:21.091807 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:21.091814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:21.091959 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:21.092088 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:21.093016 (+ 928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57566: BlockingRecv error: recv got EOF from 127.0.0.1:57566 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 02:23:21.129503 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:21.122317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39254 (local address 127.4.140.253:38709)
0928 02:23:21.122509 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:21.122516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:21.123015 (+ 499us) server_negotiation.cc:407] Connection header received
0928 02:23:21.123223 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:21.123230 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:21.123331 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:21.123443 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:21.123952 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.125165 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.127162 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.127860 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.128018 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:21.128634 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:21.128789 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:21.128881 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:21.129089 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 02:23:21.129205 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 02:23:21.140111 27691 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:21.137654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.4.140.252:42101)
0928 02:23:21.137841 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:21.137847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:21.138381 (+ 534us) server_negotiation.cc:407] Connection header received
0928 02:23:21.138578 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:21.138586 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:21.138705 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:21.138841 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:21.139830 (+ 989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57578: BlockingRecv error: recv got EOF from 127.0.0.1:57578 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:23:21.181818 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:21.175466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39256 (local address 127.4.140.253:38709)
0928 02:23:21.175778 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:21.175786 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:21.176383 (+ 597us) server_negotiation.cc:407] Connection header received
0928 02:23:21.176643 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:21.176648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:21.176733 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:21.176858 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:21.177559 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.178355 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.179737 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.180359 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.180534 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:21.180943 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:21.181052 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:21.181166 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:21.181390 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 02:23:21.181458 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 02:23:21.190035 27691 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:21.188552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57590 (local address 127.4.140.252:42101)
0928 02:23:21.188776 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:21.188783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:21.188804 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:23:21.188875 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:21.188881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:21.188981 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:21.189082 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:21.189799 (+ 717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57590: BlockingRecv error: recv got EOF from 127.0.0.1:57590 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:23:21.262622 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:21.257762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39272 (local address 127.4.140.253:38709)
0928 02:23:21.257951 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:21.257956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:21.258262 (+ 306us) server_negotiation.cc:407] Connection header received
0928 02:23:21.258455 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:21.258459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:21.258529 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:21.258621 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:21.259023 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.259784 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.261160 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.261562 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.261655 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:21.262026 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:21.262135 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:21.262199 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:21.262328 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 02:23:21.262397 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 02:23:21.270944 27691 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:23:21.268817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.4.140.252:42101)
0928 02:23:21.269061 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:21.269068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:21.269378 (+ 310us) server_negotiation.cc:407] Connection header received
0928 02:23:21.269676 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:21.269692 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:21.269769 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:21.269890 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:23:21.270703 (+ 813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57600: BlockingRecv error: recv got EOF from 127.0.0.1:57600 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:23:21.316547 27668 catalog_manager.cc:1259] Loaded cluster ID: 48f27ece03914a17bad74e81dc62e044
I20260928 02:23:21.316648 27668 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71: loading cluster ID for follower catalog manager: success
I20260928 02:23:21.319169 27668 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71: acquiring CA information for follower catalog manager: success
I20260928 02:23:21.321080 27668 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8e53da5be01d40cc8be3f23f3a580b71: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:23:21.407660 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:21.402541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39274 (local address 127.4.140.253:38709)
0928 02:23:21.402805 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:21.402812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:21.403032 (+ 220us) server_negotiation.cc:407] Connection header received
0928 02:23:21.403247 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:21.403251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:21.403331 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:21.403414 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:21.403833 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.404599 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.406003 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.406424 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.406581 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:21.406936 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:21.407053 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:21.407122 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:21.407244 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 02:23:21.407310 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260928 02:23:21.419557 27691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:21.413761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57604 (local address 127.4.140.252:42101)
0928 02:23:21.413999 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:21.414005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:21.414400 (+ 395us) server_negotiation.cc:407] Connection header received
0928 02:23:21.414594 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:21.414599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:21.414791 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:21.414912 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:21.415281 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.416063 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.417357 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.417794 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.417907 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:21.418380 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:21.418934 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:21.419043 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:21.419152 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 02:23:21.419217 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:23:21.433295 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:21.425530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39276 (local address 127.4.140.253:38709)
0928 02:23:21.425781 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:21.425791 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:21.427092 (+ 1301us) server_negotiation.cc:407] Connection header received
0928 02:23:21.427356 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:21.427365 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:21.427477 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:21.427571 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:21.428240 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.429239 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.431080 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.431772 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.431935 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:21.432498 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:21.432630 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:21.432722 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:21.432880 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 02:23:21.432969 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 02:23:21.434094 27691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:21.426763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57614 (local address 127.4.140.252:42101)
0928 02:23:21.426928 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:21.426934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:21.427793 (+ 859us) server_negotiation.cc:407] Connection header received
0928 02:23:21.428093 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:21.428098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:21.428177 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:21.428290 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:21.428821 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.429913 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.431761 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.432479 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.432604 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:21.433093 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:21.433219 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:21.433323 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:21.433780 (+ 457us) server_negotiation.cc:299] Negotiation successful
0928 02:23:21.433847 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:23:21.435048 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:23:21.425335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54010 (local address 127.4.140.254:42211)
0928 02:23:21.428030 (+ 2695us) server_negotiation.cc:206] Beginning negotiation
0928 02:23:21.428035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:23:21.428053 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:23:21.428166 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:23:21.428170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:23:21.428410 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:23:21.428613 (+ 203us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:23:21.429281 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.430261 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.432383 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:23:21.433038 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:23:21.433656 (+ 618us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:23:21.433866 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:23:21.434381 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:23:21.434487 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:23:21.434589 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 02:23:21.434670 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2532,"thread_start_us":2440,"threads_started":1}
I20260928 02:23:21.439776 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 27436
I20260928 02:23:21.441016 27498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:23:21.552546 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 27436
I20260928 02:23:21.573897 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 27510
I20260928 02:23:21.575321 27572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:23:21.685490 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 27510
I20260928 02:23:21.706364 4659 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu pid 27584
I20260928 02:23:21.707557 27646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:23:21.815572 4659 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4rqyh/build/asan/bin/kudu with pid 27584
2026-09-28T02:23:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4468 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8955 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (366424 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST